From p.f.moore at gmail.com Thu May 2 17:53:38 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 May 2013 16:53:38 +0100 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: Message-ID: On 30 April 2013 14:23, Vinay Sajip wrote: > Your feedback will be gratefully received! > One thing that does slightly bother me. If distil is invoked without the -p or -e option, it defaults to installing into the system Python. (At least on Windows, on Unix I guess it gets the Python currently first on $PATH, due to the use of /usr/bin/env). Pretty much all of the time, that's *not* what I want (at least for install/uninstall type actions). Would it be worth requiring the use of -e or -p, when running distil to install/uninstall? (There could be a --system flag to say "yes, I really do want the system Python" if that's worth having as well as an explicit -p). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu May 2 18:11:08 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 2 May 2013 17:11:08 +0100 (BST) Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: Message-ID: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> > From: Paul Moore >One thing that does slightly bother me. If distil is invoked without the -p or -e option, it defaults to installing into the system Python. That's not supposed to happen - it's supposed to install to the user site (PEP 370 Per user site-packages directory) unless you specify --system. It will use the system Python unless you specify -p. So: distil install xyz should install xyz to the per-user site-packages using the default Python, say 2.7. distil -p python3.2 install xyz should install xyz to the per-user site-packages, using Python 3.2. Specifying --system should install to the system-wide Python site-packages for the running Python. In a venv, installations should always be to the venv. I will see if the problem is Windows specific - the behaviour seems OK on POSIX. Regards, Vinay Sajip From p.f.moore at gmail.com Thu May 2 19:07:23 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 May 2013 18:07:23 +0100 Subject: [Distutils] distil 0.1.1 released In-Reply-To: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: Sorry, I knew that it installed to the per-user site packages directory, but I never use that so I don't fully understand the implications. But nevertheless, for my current usage, distil install foo is 99% of the time a user error, as I maintain a completely clean system Python, and do all my package installs in virtualenvs. Maybe the per-user site packages is something I should be using, and if I do, then I can afford to be less obsessive about keeping my system Python clean. But until distil became available, I haven't had the tools to work with the user site packages directory, so it's new ground for me. The one thing that *is* different between Unix and Windows is that on Unix, the hashbang line of #!/usr/bin/env python will use the currently active Python (system or virtualenv as appropriate) but under Windows it will always use the system Python (2 or 3 depending on py.ini). Maybe the launcher should treat /sur/bin/env python differently, and try looking on PATH before using the default in that case, but that's not how it works, sadly (indeed, there's no way of getting that behaviour with the current launcher). I can imagine having a virtualenv active and running distil install foo and being surprised that the install doesn't go to the active virtualenv. Paul On 2 May 2013 17:11, Vinay Sajip wrote: > > From: Paul Moore > >One thing that does slightly bother me. If distil is invoked without the > -p or -e option, it defaults to installing into the system Python. > > That's not supposed to happen - it's supposed to install to the user site > (PEP 370 Per user site-packages directory) unless you specify --system. It > will use the system Python unless you specify -p. So: > > distil install xyz should install xyz to the per-user site-packages using > the default Python, say 2.7. > distil -p python3.2 install xyz should install xyz to the per-user > site-packages, using Python 3.2. > > > Specifying --system should install to the system-wide Python site-packages > for the running Python. > > In a venv, installations should always be to the venv. > > > I will see if the problem is Windows specific - the behaviour seems OK on > POSIX. > > Regards, > > Vinay Sajip > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu May 2 19:29:12 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 2 May 2013 18:29:12 +0100 (BST) Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: <1367515752.7258.YahooMailNeo@web171406.mail.ir2.yahoo.com> > From: Paul Moore > But nevertheless, for my current usage, distil install foo is 99% of the time a > user error, as I maintain a completely clean system Python, and do all my > package installs in virtualenvs. Maybe the per-user site packages is something > I should be using, and if I do, then I can afford to be less obsessive about keeping > my system Python clean. But until distil became available, I haven't had the tools to > work with the user site packages directory, so it's new ground for me. If you do all your package installs into venvs, then isn't "distil -e venvdir install foo" enough? You don't need to activate the venv, just tell distil which venv to install into via the -e. > The one thing that *is* different between Unix and Windows is that on Unix, the > hashbang line of #!/usr/bin/env python will use the currently active Python (system or > virtualenv as appropriate) but under Windows it will always use the system Python (2 > or 3 depending on py.ini). Maybe the launcher should treat /sur/bin/env python differently, > and try looking on PATH before using the default in that case, but that's not how it works, > sadly (indeed, there's no way of getting that behaviour with the current launcher). I can > imagine having a virtualenv active and running distil install foo and being surprised that the > install doesn't go to the active virtualenv. I have implemented path searching in the launcher, but it's currently disabled. I can re-enable it (such that the path is only searched if the PYLAUNCH_SEARCHPATH environment variable is set) and create a new release of the standalone launcher - are you willing to be a guinea pig for this functionality? Regards, Vinay Sajip From mal at egenix.com Thu May 2 19:41:34 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 May 2013 19:41:34 +0200 Subject: [Distutils] HTTPS and certificate check update for distribute ? In-Reply-To: <517940EA.1020908@egenix.com> References: <517940EA.1020908@egenix.com> Message-ID: <5182A54E.7010800@egenix.com> On 25.04.2013 16:42, M.-A. Lemburg wrote: > The latest pip supports HTTPS URLs and certificate checks > (according to the change log). > > Will there be a release of distribute that implements the > same changes ? > > The current 0.6.36 still defaults to the HTTP PyPI address > and doesn't do certificate checks. Hmm, given the lack of response, I guess this will take a little longer ;-) I've put Tarek on CC. Perhaps he can chime in... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 02 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Thu May 2 21:21:54 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 May 2013 20:21:54 +0100 Subject: [Distutils] distil 0.1.1 released In-Reply-To: <1367515752.7258.YahooMailNeo@web171406.mail.ir2.yahoo.com> References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1367515752.7258.YahooMailNeo@web171406.mail.ir2.yahoo.com> Message-ID: On 2 May 2013 18:29, Vinay Sajip wrote: > If you do all your package installs into venvs, then isn't "distil -e > venvdir install foo" enough? You don't need to activate the venv, just tell > distil which venv > to install into via the -e. > It's not that -e is insufficient, rather that *forgetting* the -e doesn't fail, but rather does the wrong thing silently (and has an unintended "global" impact). If distil used the "active" Python by default (which it does on Unix) then this wouldn't concern me so much. But it is a small point, I'd admit. > I have implemented path searching in the launcher, but it's currently > disabled. I can re-enable it (such that the path is only searched if the > PYLAUNCH_SEARCHPATH environment variable is set) and create a new release > of the standalone launcher - are you willing to be a guinea pig for this > functionality? > What I would like is for #!/usr/bin/env python to be treated specially as a path search (it's the only case on Unix that does a path search after all), with a fallback of using the default python (like #!python) if there is no python on $PATH. That completely replicates the Unix behaviour (i.e., the author's intention, I would assume). I would argue that this does not need to be conditionally enabled, but given that it deviates from the PEP I can live with the requirement. Although I'd prefer it if the override was via the ini file rather than being via an environment variable. Oh, and can I have a pony? :-) Seriously, I'd be happy to test this. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri May 3 00:48:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 May 2013 08:48:33 +1000 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: On 3 May 2013 03:09, "Paul Moore" wrote: > > Sorry, I knew that it installed to the per-user site packages directory, but I never use that so I don't fully understand the implications. > > But nevertheless, for my current usage, distil install foo is 99% of the time a user error, as I maintain a completely clean system Python, and do all my package installs in virtualenvs. Maybe the per-user site packages is something I should be using, and if I do, then I can afford to be less obsessive about keeping my system Python clean. But until distil became available, I haven't had the tools to work with the user site packages directory, so it's new ground for me. "pip --user" manipulates the per-user directories (and I believe I filed a ticket a while back suggesting it should migrate to the model distil now uses). Cheers, Nick. > > The one thing that *is* different between Unix and Windows is that on Unix, the hashbang line of #!/usr/bin/env python will use the currently active Python (system or virtualenv as appropriate) but under Windows it will always use the system Python (2 or 3 depending on py.ini). Maybe the launcher should treat /sur/bin/env python differently, and try looking on PATH before using the default in that case, but that's not how it works, sadly (indeed, there's no way of getting that behaviour with the current launcher). I can imagine having a virtualenv active and running distil install foo and being surprised that the install doesn't go to the active virtualenv. > > Paul > > > On 2 May 2013 17:11, Vinay Sajip wrote: >> >> > From: Paul Moore >> >One thing that does slightly bother me. If distil is invoked without the -p or -e option, it defaults to installing into the system Python. >> >> That's not supposed to happen - it's supposed to install to the user site (PEP 370 Per user site-packages directory) unless you specify --system. It will use the system Python unless you specify -p. So: >> >> distil install xyz should install xyz to the per-user site-packages using the default Python, say 2.7. >> distil -p python3.2 install xyz should install xyz to the per-user site-packages, using Python 3.2. >> >> >> Specifying --system should install to the system-wide Python site-packages for the running Python. >> >> In a venv, installations should always be to the venv. >> >> >> I will see if the problem is Windows specific - the behaviour seems OK on POSIX. >> >> Regards, >> >> Vinay Sajip >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri May 3 01:17:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 May 2013 09:17:54 +1000 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: On 3 May 2013 09:08, "Donald Stufft" wrote: > > On May 2, 2013, at 6:48 PM, Nick Coghlan wrote: >> >> "pip --user" manipulates the per-user directories (and I believe I filed a ticket a while back suggesting it should migrate to the model distil now uses). >> >> Cheers, >> Nick. > > I don't think it makes sense for pip to do this tbh. Language level installers should leave "/usr" alone by default, as that part of the filesystem is the domain of system packages. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 3 01:08:18 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 2 May 2013 19:08:18 -0400 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: On May 2, 2013, at 6:48 PM, Nick Coghlan wrote: > "pip --user" manipulates the per-user directories (and I believe I filed a ticket a while back suggesting it should migrate to the model distil now uses). > > Cheers, > Nick. > I don't think it makes sense for pip to do this tbh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 3 01:30:12 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 2 May 2013 19:30:12 -0400 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> On May 2, 2013, at 7:17 PM, Nick Coghlan wrote: > > On 3 May 2013 09:08, "Donald Stufft" wrote: > > > > On May 2, 2013, at 6:48 PM, Nick Coghlan wrote: > >> > >> "pip --user" manipulates the per-user directories (and I believe I filed a ticket a while back suggesting it should migrate to the model distil now uses). > >> > >> Cheers, > >> Nick. > > > > I don't think it makes sense for pip to do this tbh. > > Language level installers should leave "/usr" alone by default, as that part of the filesystem is the domain of system packages. > > Cheers, > Nick. Out of the three primary OSs only one of them has a concept of system packages. Regardless it's a major change in behavior and isn't a clear cut right or wrong choice. It's going to need a stronger justification than that to break backwards compatibility in such a large way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri May 3 03:02:38 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 May 2013 11:02:38 +1000 Subject: [Distutils] distil 0.1.1 released In-Reply-To: <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> Message-ID: On Fri, May 3, 2013 at 9:30 AM, Donald Stufft wrote: > > On May 2, 2013, at 7:17 PM, Nick Coghlan wrote: > > > On 3 May 2013 09:08, "Donald Stufft" wrote: >> >> On May 2, 2013, at 6:48 PM, Nick Coghlan wrote: >>> >>> "pip --user" manipulates the per-user directories (and I believe I filed >>> a ticket a while back suggesting it should migrate to the model distil now >>> uses). >>> >>> Cheers, >>> Nick. >> >> I don't think it makes sense for pip to do this tbh. > > Language level installers should leave "/usr" alone by default, as that part > of the filesystem is the domain of system packages. > > Cheers, > Nick. > > Out of the three primary OSs only one of them has a concept of system > packages. Regardless it's a major change in behavior and isn't a clear cut > right or wrong choice. It's going to need a stronger justification than that > to break backwards compatibility in such a large way. How about: I won't approve PEP 439 as long as pip still writes to the system site-packages by default when running as an ordinary user on Linux systems? "sudo pip" needs to be banished from the command line lexicon of Python developers. The transition isn't that hard: add --system now, switch the default from --system to --user for anyone *other than root* some time before Python 3.4. Anyone using virtual environments or an explicit --user or --system will be utterly unaffected when the default changes, as will anyone running an explicit "sudo pip" (in order to get write access to the system Python site-packages). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Fri May 3 03:39:18 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 2 May 2013 21:39:18 -0400 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> Message-ID: <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> On May 2, 2013, at 9:02 PM, Nick Coghlan wrote: > On Fri, May 3, 2013 at 9:30 AM, Donald Stufft wrote: >> >> On May 2, 2013, at 7:17 PM, Nick Coghlan wrote: >> >> >> On 3 May 2013 09:08, "Donald Stufft" wrote: >>> >>> On May 2, 2013, at 6:48 PM, Nick Coghlan wrote: >>>> >>>> "pip --user" manipulates the per-user directories (and I believe I filed >>>> a ticket a while back suggesting it should migrate to the model distil now >>>> uses). >>>> >>>> Cheers, >>>> Nick. >>> >>> I don't think it makes sense for pip to do this tbh. >> >> Language level installers should leave "/usr" alone by default, as that part >> of the filesystem is the domain of system packages. >> >> Cheers, >> Nick. >> >> Out of the three primary OSs only one of them has a concept of system >> packages. Regardless it's a major change in behavior and isn't a clear cut >> right or wrong choice. It's going to need a stronger justification than that >> to break backwards compatibility in such a large way. > > How about: I won't approve PEP 439 as long as pip still writes to the > system site-packages by default when running as an ordinary user on > Linux systems? "sudo pip" needs to be banished from the command line > lexicon of Python developers. > > The transition isn't that hard: add --system now, switch the default > from --system to --user for anyone *other than root* some time before > Python 3.4. Anyone using virtual environments or an explicit --user or > --system will be utterly unaffected when the default changes, as will > anyone running an explicit "sudo pip" (in order to get write access to > the system Python site-packages). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia So your answer to my wanting stronger justification is "because I said so"? *Why* does ``sudo pip`` need to be banished? ``pip install --user`` is not any safer, it's not any more or less wrong. It's an option. If pip was brand new I'd probably be of the opinion that I prefer the user scheme by default. However pip is not new, and it has a long history of working in this way. Changes can be made but I'm loath to support a large change in behavior like this without strong justification as to *why*. I also think your proposed solution of --user being the default unless we are running as root is suboptimal. Conditional defaults are confusing to end users, even more so when a common "error" condition (e.g. I forgot to type sudo before I pip installed, or I forgot to activate my virtualenv) is going to move from being an error to installing to the user scheme and *working*? right up until the time you actually try and use it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri May 3 12:29:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 May 2013 20:29:02 +1000 Subject: [Distutils] distil 0.1.1 released In-Reply-To: <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> Message-ID: On 3 May 2013 11:39, "Donald Stufft" wrote: > > > On May 2, 2013, at 9:02 PM, Nick Coghlan wrote: > > > On Fri, May 3, 2013 at 9:30 AM, Donald Stufft wrote: > >> > >> On May 2, 2013, at 7:17 PM, Nick Coghlan wrote: > >> > >> > >> On 3 May 2013 09:08, "Donald Stufft" wrote: > >>> > >>> On May 2, 2013, at 6:48 PM, Nick Coghlan wrote: > >>>> > >>>> "pip --user" manipulates the per-user directories (and I believe I filed > >>>> a ticket a while back suggesting it should migrate to the model distil now > >>>> uses). > >>>> > >>>> Cheers, > >>>> Nick. > >>> > >>> I don't think it makes sense for pip to do this tbh. > >> > >> Language level installers should leave "/usr" alone by default, as that part > >> of the filesystem is the domain of system packages. > >> > >> Cheers, > >> Nick. > >> > >> Out of the three primary OSs only one of them has a concept of system > >> packages. Regardless it's a major change in behavior and isn't a clear cut > >> right or wrong choice. It's going to need a stronger justification than that > >> to break backwards compatibility in such a large way. > > > > How about: I won't approve PEP 439 as long as pip still writes to the > > system site-packages by default when running as an ordinary user on > > Linux systems? "sudo pip" needs to be banished from the command line > > lexicon of Python developers. > > > > The transition isn't that hard: add --system now, switch the default > > from --system to --user for anyone *other than root* some time before > > Python 3.4. Anyone using virtual environments or an explicit --user or > > --system will be utterly unaffected when the default changes, as will > > anyone running an explicit "sudo pip" (in order to get write access to > > the system Python site-packages). > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > So your answer to my wanting stronger justification is "because I said so"? Because sysadmins don't participate in upstream development and non-platform provided tools messing with /usr is wrong, wrong, wrong. The system Python is the domain of distro tools, we should be leaving it the hell alone. Cheers, Nick. > > *Why* does ``sudo pip`` need to be banished? ``pip install --user`` is not any safer, it's not > any more or less wrong. It's an option. If pip was brand new I'd probably be of the opinion > that I prefer the user scheme by default. However pip is not new, and it has a long history > of working in this way. Changes can be made but I'm loath to support a large change in > behavior like this without strong justification as to *why*. > > I also think your proposed solution of --user being the default unless we are running as root > is suboptimal. Conditional defaults are confusing to end users, even more so when a common > "error" condition (e.g. I forgot to type sudo before I pip installed, or I forgot to activate my virtualenv) > is going to move from being an error to installing to the user scheme and *working*? right up until > the time you actually try and use it. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 3 13:10:03 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 May 2013 07:10:03 -0400 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> Message-ID: On May 3, 2013, at 6:29 AM, Nick Coghlan wrote: > Because sysadmins don't participate in upstream development and non-platform provided tools messing with /usr is wrong, wrong, wrong. > > The system Python is the domain of distro tools, we should be leaving it the hell alone. > > Cheers, > Nick. > To be honest your proposal doesn't really do that since you want it to install into the site-packages if they are running it as root. If that's you're argument you should at least be consistent within your own proposal. One of my main problems with your proposal is that we suddenly have surprising behavior about where packages are being installed to: Linux + root user = site-packages Linux + regular user = user-packages (<-- Going to confuse people because it's different) Linux + regular user + virtualenv = site-packages I don't know much about RHEL, but Debian "installing globally" means installing into /usr/local/lib/pythonX.Y/site-packages/ which is exactly where Debian wants you to put python libraries not installed via apt. Apt drops things into /usr/local/lib/pythonX.Y/dist-packages. Even if installing into the "system" python globally is wrong is there even a way too determine if we are attempting to install into a *system* python or not? I could be wrong but afaik there's not a good way to determine if this is a python that is managed by the system or one that was installed otherwise. If that's the case then we break by default one of the viable methods of isolation *from* system. This is very popular on OSX where people tend to install a non system python via homebrew to isolate themselves. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 3 14:23:49 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 May 2013 13:23:49 +0100 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> Message-ID: > > How about: I won't approve PEP 439 as long as pip still writes to the > > system site-packages by default when running as an ordinary user on > > Linux systems? "sudo pip" needs to be banished from the command line > > lexicon of Python developers. What is the proposal for Windows here? Would the default behaviour change? There's no concept of "sudo" on Windows. Personally, I don't care much as I only use pip in virtualenvs, but backward compatibility *is* important, even if the current behaviour is considered wrong. Also, don't forget that the current release of pip supports Python 2.5, which doesn't have per-user site-packages. The next release will drop support for 2.5, so we *could* start the switch to the user directory at the same time, but up until now that wasn't an option. PEP 439 may be about Python 3.4, but Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri May 3 20:14:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 May 2013 04:14:40 +1000 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> Message-ID: I would also be relatively happy for pip to refuse the temptation to guess if run globally and require an explicit --user or --system whenever it is run outside a virtual environment. However, I think it's better to make the typical "pip install whatever" work for most unprivileged users without requiring elevated privileges. I agree the proposed exception for root doesn't make sense so I withdraw that idea, even though installing things into root's home directory is a little strange. As far as Debian's dist-packages setup goes, that's their workaround for this misfeature of the current Python packaging ecosystem. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 3 20:23:28 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 May 2013 14:23:28 -0400 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> Message-ID: <94906541-5A26-4160-8B81-539A1758337A@stufft.io> On May 3, 2013, at 2:14 PM, Nick Coghlan wrote: > I would also be relatively happy for pip to refuse the temptation to guess if run globally and require an explicit --user or --system whenever it is run outside a virtual environment. However, I think it's better to make the typical "pip install whatever" work for most unprivileged users without requiring elevated privileges. > > I agree the proposed exception for root doesn't make sense so I withdraw that idea, even though installing things into root's home directory is a little strange. > > As far as Debian's dist-packages setup goes, that's their workaround for this misfeature of the current Python packaging ecosystem. > > Cheers, > Nick. > How do you determine "run globally"? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Fri May 3 20:24:55 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Fri, 3 May 2013 11:24:55 -0700 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> Message-ID: <67ADC1E0-7184-4BDF-AFF0-6A07CEF6E976@coderanger.net> On May 3, 2013, at 11:14 AM, Nick Coghlan wrote: > I would also be relatively happy for pip to refuse the temptation to guess if run globally and require an explicit --user or --system whenever it is run outside a virtual environment. However, I think it's better to make the typical "pip install whatever" work for most unprivileged users without requiring elevated privileges. > > I agree the proposed exception for root doesn't make sense so I withdraw that idea, even though installing things into root's home directory is a little strange. > > As far as Debian's dist-packages setup goes, that's their workaround for this misfeature of the current Python packaging ecosystem. As someone responsible for working with Python app deployment tools, this _will_ break the universe. Yes people should be using virtualenv, however many don't, deal with it. People expect "package install" as root to work like every other package system (yum, apt, take your pick) where they run "sudo pip install a b c d" and then run their app as a service-specific user. Magically writing to ~root will clearly not work in this case unless you also run your app as root (though I know some people do that too, but not behavior that should be encouraged). This proposal is entirely non-viable for anything but 100% best-practices users, full stop. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tseaver at palladion.com Fri May 3 20:25:02 2013 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 03 May 2013 14:25:02 -0400 Subject: [Distutils] distil 0.1.1 released In-Reply-To: <67ADC1E0-7184-4BDF-AFF0-6A07CEF6E976@coderanger.net> References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> <67ADC1E0-7184-4BDF-AFF0-6A07CEF6E976@coderanger.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/03/2013 02:24 PM, Noah Kantrowitz wrote: > People expect "package install" as root to work like every other > package system (yum, apt, take your pick) where they run "sudo pip > install a b c d" and then run their app as a service-specific user. I don't know any of those people: either folks have learned *not* to install as root, or else they run as root too. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGEAP4ACgkQ+gerLs4ltQ6odACfXcO7KHBYvPJhWOZCXMtMTZsn bxIAn3j9jwRe70de+O0mIY4K2kfDrfzs =WgtN -----END PGP SIGNATURE----- From donald at stufft.io Fri May 3 20:37:40 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 May 2013 14:37:40 -0400 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> <67ADC1E0-7184-4BDF-AFF0-6A07CEF6E976@coderanger.net> Message-ID: <885B7E98-D02A-40D1-B0DE-32242166D1E6@stufft.io> On May 3, 2013, at 2:25 PM, Tres Seaver wrote: > Signed PGP part > On 05/03/2013 02:24 PM, Noah Kantrowitz wrote: > > > People expect "package install" as root to work like every other > > package system (yum, apt, take your pick) where they run "sudo pip > > install a b c d" and then run their app as a service-specific user. > > I don't know any of those people: either folks have learned *not* to > install as root, or else they run as root too. > I know a fair number of people like that. The common wisdom is *not* to install python packages from the system package mangers. because they generally either don't have everything so you'll need to install from PyPI anyways, or they'll have really old versions. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From carl at oddbird.net Sat May 4 00:37:52 2013 From: carl at oddbird.net (Carl Meyer) Date: Fri, 03 May 2013 16:37:52 -0600 Subject: [Distutils] distil 0.1.1 released In-Reply-To: References: <1367511068.37637.YahooMailNeo@web171402.mail.ir2.yahoo.com> <46353D00-F683-4909-B21A-CE97FAF24C43@stufft.io> <85C9FD1D-0BF7-4BDD-9AE1-6C54DEE591E3@stufft.io> Message-ID: <51843C40.7060803@oddbird.net> Hi Nick On 05/03/2013 12:14 PM, Nick Coghlan wrote: > I would also be relatively happy for pip to refuse the temptation to > guess if run globally and require an explicit --user or --system > whenever it is run outside a virtual environment. However, I think it's > better to make the typical "pip install whatever" work for most > unprivileged users without requiring elevated privileges. > > I agree the proposed exception for root doesn't make sense so I withdraw > that idea, even though installing things into root's home directory is a > little strange. > > As far as Debian's dist-packages setup goes, that's their workaround for > this misfeature of the current Python packaging ecosystem. IMO requiring either --user or --system is bad UI (we should have a sane default). Defaulting to --user is tempting, but also a bad idea, both because of the massive adjustment that it would require from existing users, and because it does the wrong thing in many cases where there are multiple Pythons compiled on the system. I think the best solution to this problem is quite easily implemented and non-intrusive: an improved error message when pip finds that it lacks permission to install to site-packages. This error message could recommend --user and/or virtualenv, could warn quite strongly against installing to this location unless you really know what you are doing, etc. Also, I think that for the specific situation of a Linux distribution's "system Python" (which, as Donald has pointed out, is only one mode of Python installation out of many), Debian's solution is better than just a workaround, it is conceptually more or less the Right Thing: separate distro-managed Python code from user-managed Python code, without preventing independent installation of Python-installation-wide code by the user. (Though some aspects of the implementation of Debian's solution are sub-optimal and have required ugly workarounds in pip. It would be better if multiple site-packages were a first-class concept in Python's own site.py so Debian's solution could be implemented via configuration rather than patches.) Carl From ncoghlan at gmail.com Sat May 4 03:47:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 May 2013 11:47:27 +1000 Subject: [Distutils] PEP 439 and default install locations (was Re: distil 0.1.1 released) Message-ID: On 4 May 2013 09:03, "Carl Meyer" wrote: > > Hi Nick > > On 05/03/2013 12:14 PM, Nick Coghlan wrote: > > I would also be relatively happy for pip to refuse the temptation to > > guess if run globally and require an explicit --user or --system > > whenever it is run outside a virtual environment. However, I think it's > > better to make the typical "pip install whatever" work for most > > unprivileged users without requiring elevated privileges. > > > > I agree the proposed exception for root doesn't make sense so I withdraw > > that idea, even though installing things into root's home directory is a > > little strange. > > > > As far as Debian's dist-packages setup goes, that's their workaround for > > this misfeature of the current Python packaging ecosystem. > > IMO requiring either --user or --system is bad UI (we should have a sane > default). Defaulting to --user is tempting, but also a bad idea, both > because of the massive adjustment that it would require from existing > users, and because it does the wrong thing in many cases where there are > multiple Pythons compiled on the system. > > I think the best solution to this problem is quite easily implemented > and non-intrusive: an improved error message when pip finds that it > lacks permission to install to site-packages. This error message could > recommend --user and/or virtualenv, could warn quite strongly against > installing to this location unless you really know what you are doing, etc. That sounds eminently reasonable. > Also, I think that for the specific situation of a Linux distribution's > "system Python" (which, as Donald has pointed out, is only one mode of > Python installation out of many), Debian's solution is better than just > a workaround, it is conceptually more or less the Right Thing: separate > distro-managed Python code from user-managed Python code, without > preventing independent installation of Python-installation-wide code by > the user. > > (Though some aspects of the implementation of Debian's solution are > sub-optimal and have required ugly workarounds in pip. It would be > better if multiple site-packages were a first-class concept in Python's > own site.py so Debian's solution could be implemented via configuration > rather than patches.) Agreed, blessing something like Debian's solution sounds like a better idea than changing pip's default installation location. To provide that blessing, I think PEP 439 should be expanded to explicitly cover this aspect (there's another subtlety, in that it may be desirable for distro packages to be available when running with -S - I'm not sure how Debian currently handle that). Essentially, my goal is to have a clearly defined, consistent, cross-distro-and-platform shadowing behaviour for cases where you install a Python software distribution from a system package manager *and* through pip. The current situation, where the system tools and the Python specific tools tread on each other's toes at the filesystem layer (unless you change the default Python path, as Debian does) isn't appropriate for something we are going to ship as part of the standard Python toolkit. (Coincidentally, this may lead to my CPython bootstrapping changes becoming packaging related!) Cheers, Nick. > > Carl > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat May 4 16:12:15 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 4 May 2013 10:12:15 -0400 Subject: [Distutils] PEP 439 and default install locations (was Re: distil 0.1.1 released) In-Reply-To: References: Message-ID: <3A0BFC2B-8999-4F41-83BE-1182C9C11399@stufft.io> On May 3, 2013, at 9:47 PM, Nick Coghlan wrote: > Agreed, blessing something like Debian's solution sounds like a better idea than changing pip's default installation location. To provide that blessing, I think PEP 439 should be expanded to explicitly cover this aspect (there's another subtlety, in that it may be desirable for distro packages to be available when running with -S - I'm not sure how Debian currently handle that). > > Essentially, my goal is to have a clearly defined, consistent, cross-distro-and-platform shadowing behaviour for cases where you install a Python software distribution from a system package manager *and* through pip. The current situation, where the system tools and the Python specific tools tread on each other's toes at the filesystem layer (unless you change the default Python path, as Debian does) isn't appropriate for something we are going to ship as part of the standard Python toolkit. > > (Coincidentally, this may lead to my CPython bootstrapping changes becoming packaging related!) > > Cheers, > Nick. > This sounds excellent to me. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Sun May 5 00:28:57 2013 From: pje at telecommunity.com (PJ Eby) Date: Sat, 4 May 2013 18:28:57 -0400 Subject: [Distutils] HTTPS and certificate check update for distribute ? In-Reply-To: <5182A54E.7010800@egenix.com> References: <517940EA.1020908@egenix.com> <5182A54E.7010800@egenix.com> Message-ID: On Thu, May 2, 2013 at 1:41 PM, M.-A. Lemburg wrote: > On 25.04.2013 16:42, M.-A. Lemburg wrote: >> The latest pip supports HTTPS URLs and certificate checks >> (according to the change log). >> >> Will there be a release of distribute that implements the >> same changes ? >> >> The current 0.6.36 still defaults to the HTTP PyPI address >> and doesn't do certificate checks. FWIW, I've just checked in the first phase of my SSL implementation for setuptools, to the repository that Jason is doing merges from. The current implementation silently uses system-wide root certs from the Windows registry or from *nixes that have a well-known root bundle location. (But won't find anything on OS X by default). It also doesn't have any command-line options yet to explicitly select the certs used or to control SSL verification. But it does offer the ability to "easy_install setuptools[ssl]" to download verified copies of all the dependencies needed to get SSL support in earlier Pythons, including win32 binaries where applicable, without needing anything but the original setuptools distribution needing to have been downloaded manually via SSL. There is still more that needs to be done besides command-line options, warnings, and docs; providing default root certs for OS X, for example. I've got a couple different ideas on that, from bundling the StartCom root cert that python.org uses, to creating a separate ca_bundle distribution that contains the files. There's another interesting gotcha with OS X certs, which is that the platform-provided openssl may check its built-in cert store in addition to what you give it explicitly, which could be a problem. In short: providing practical, cross-platform, cross-wide-array-of-python-versions SSL support is *hard*. I'm not too surprised you haven't heard from anybody yet. ;-) From donald at stufft.io Sun May 5 02:01:13 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 4 May 2013 20:01:13 -0400 Subject: [Distutils] HTTPS and certificate check update for distribute ? In-Reply-To: References: <517940EA.1020908@egenix.com> <5182A54E.7010800@egenix.com> Message-ID: <1DD30A72-EE31-4EC7-B1BF-C2A51D4FE61B@stufft.io> On May 4, 2013, at 6:28 PM, PJ Eby wrote: > On Thu, May 2, 2013 at 1:41 PM, M.-A. Lemburg wrote: >> On 25.04.2013 16:42, M.-A. Lemburg wrote: >>> The latest pip supports HTTPS URLs and certificate checks >>> (according to the change log). >>> >>> Will there be a release of distribute that implements the >>> same changes ? >>> >>> The current 0.6.36 still defaults to the HTTP PyPI address >>> and doesn't do certificate checks. > > FWIW, I've just checked in the first phase of my SSL implementation > for setuptools, to the repository that Jason is doing merges from. > The current implementation silently uses system-wide root certs from > the Windows registry or from *nixes that have a well-known root bundle > location. (But won't find anything on OS X by default). It also > doesn't have any command-line options yet to explicitly select the > certs used or to control SSL verification. But it does offer the > ability to "easy_install setuptools[ssl]" to download verified copies > of all the dependencies needed to get SSL support in earlier Pythons, > including win32 binaries where applicable, without needing anything > but the original setuptools distribution needing to have been > downloaded manually via SSL. > > There is still more that needs to be done besides command-line > options, warnings, and docs; providing default root certs for OS X, > for example. I've got a couple different ideas on that, from bundling > the StartCom root cert that python.org uses, to creating a separate > ca_bundle distribution that contains the files. There's another > interesting gotcha with OS X certs, which is that the > platform-provided openssl may check its built-in cert store in > addition to what you give it explicitly, which could be a problem. > > In short: providing practical, cross-platform, > cross-wide-array-of-python-versions SSL support is *hard*. I'm not > too surprised you haven't heard from anybody yet. ;-) > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig The easiest way is to just bundle the mozilla certs. They do all the work of keeping it up to date and validating them. If you're only supporting PYthon 2.6+ you can also just bundle requests and let it's cert stuff work. I think this is the ideal way if you're 2.6+ because it centralizes the codebase for dealing with SSL. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tarek at ziade.org Sun May 5 11:50:17 2013 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sun, 05 May 2013 11:50:17 +0200 Subject: [Distutils] distribute-0.6.37 In-Reply-To: <8D23CD2A18EB40B1BEA2E3CB1BEC087A@rzn.co.il> References: <8D23CD2A18EB40B1BEA2E3CB1BEC087A@rzn.co.il> Message-ID: <51862B59.6090400@ziade.org> On 5/5/13 10:57 AM, Guy Rozendorn wrote: > Hey Tarek, > > easy_install on Windows is broken in 0.6.37, see https://bitbucket.org/tarek/distribute/issue/371/launcher-manifest-is-not-installed-into > Waiting for your approval to release 0.6.38 with a fix for this issue > > Thanks, > Guy > > Hi Guy Jason & other folks are maintaining Distribute now. I cc'ed him and distutils@ Thanks Tarek -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From pje at telecommunity.com Sun May 5 20:48:49 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 5 May 2013 14:48:49 -0400 Subject: [Distutils] HTTPS and certificate check update for distribute ? In-Reply-To: <1DD30A72-EE31-4EC7-B1BF-C2A51D4FE61B@stufft.io> References: <517940EA.1020908@egenix.com> <5182A54E.7010800@egenix.com> <1DD30A72-EE31-4EC7-B1BF-C2A51D4FE61B@stufft.io> Message-ID: On Sat, May 4, 2013 at 8:01 PM, Donald Stufft wrote: > The easiest way is to just bundle the mozilla certs. They do all the work of keeping it up to date and validating them. Actually, it turns out that there's already a separately-distributed, separately licensed PyPI package for this: https://pypi.python.org/pypi/certifi So I've just added support for it to my implementation, so that if the Windows registry or well-known systemwide locations aren't available, it'll fall back automatically. So at this point about all that's left is docs and command-line options. I was previously intending to make this stuff part of an 0.6c12 release, but at this point Jason seems to be finishing up most of the merge work, and this new stuff might need some beta testing anyway, so probably it'll actually land in post-merge 0.7b stuff. > If you're only supporting PYthon 2.6+ If that were the case, I'd have been done a *long* time ago. ;-) (Setuptools still supports Python 2.3.) From donald at stufft.io Sun May 5 20:55:17 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 5 May 2013 14:55:17 -0400 Subject: [Distutils] HTTPS and certificate check update for distribute ? In-Reply-To: References: <517940EA.1020908@egenix.com> <5182A54E.7010800@egenix.com> <1DD30A72-EE31-4EC7-B1BF-C2A51D4FE61B@stufft.io> Message-ID: <1100AD51-5559-4D5C-B38F-21A495494B37@stufft.io> On May 5, 2013, at 2:48 PM, PJ Eby wrote: > On Sat, May 4, 2013 at 8:01 PM, Donald Stufft wrote: >> The easiest way is to just bundle the mozilla certs. They do all the work of keeping it up to date and validating them. > > Actually, it turns out that there's already a separately-distributed, > separately licensed PyPI package for this: > https://pypi.python.org/pypi/certifi I'm not sure if Kenneth is keeping certifi up to date anymore because he made certifi for bundling the mozilla certs with requests and has since started to bundle them directly. If he's not we might want to find someone to take it over if he'll give it up if setuptools is going to use it. License shouldn't be an issue though because the MPL is a per file not per project license FWIW. I've included Kenneth to see if he is or has any plans to keep certifi up to date. > > So I've just added support for it to my implementation, so that if the > Windows registry or well-known systemwide locations aren't available, > it'll fall back automatically. So at this point about all that's left > is docs and command-line options. > > I was previously intending to make this stuff part of an 0.6c12 > release, but at this point Jason seems to be finishing up most of the > merge work, and this new stuff might need some beta testing anyway, so > probably it'll actually land in post-merge 0.7b stuff. > > >> If you're only supporting PYthon 2.6+ > > If that were the case, I'd have been done a *long* time ago. ;-) > (Setuptools still supports Python 2.3.) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Sun May 5 23:10:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 05 May 2013 23:10:14 +0200 Subject: [Distutils] HTTPS and certificate check update for distribute ? In-Reply-To: References: <517940EA.1020908@egenix.com> <5182A54E.7010800@egenix.com> Message-ID: <5186CAB6.8080203@egenix.com> On 05.05.2013 00:28, PJ Eby wrote: > On Thu, May 2, 2013 at 1:41 PM, M.-A. Lemburg wrote: >> On 25.04.2013 16:42, M.-A. Lemburg wrote: >>> The latest pip supports HTTPS URLs and certificate checks >>> (according to the change log). >>> >>> Will there be a release of distribute that implements the >>> same changes ? >>> >>> The current 0.6.36 still defaults to the HTTP PyPI address >>> and doesn't do certificate checks. > > FWIW, I've just checked in the first phase of my SSL implementation > for setuptools, to the repository that Jason is doing merges from. > The current implementation silently uses system-wide root certs from > the Windows registry or from *nixes that have a well-known root bundle > location. (But won't find anything on OS X by default). It also > doesn't have any command-line options yet to explicitly select the > certs used or to control SSL verification. But it does offer the > ability to "easy_install setuptools[ssl]" to download verified copies > of all the dependencies needed to get SSL support in earlier Pythons, > including win32 binaries where applicable, without needing anything > but the original setuptools distribution needing to have been > downloaded manually via SSL. > > There is still more that needs to be done besides command-line > options, warnings, and docs; providing default root certs for OS X, > for example. I've got a couple different ideas on that, from bundling > the StartCom root cert that python.org uses, to creating a separate > ca_bundle distribution that contains the files. There's another > interesting gotcha with OS X certs, which is that the > platform-provided openssl may check its built-in cert store in > addition to what you give it explicitly, which could be a problem. > > In short: providing practical, cross-platform, > cross-wide-array-of-python-versions SSL support is *hard*. I'm not > too surprised you haven't heard from anybody yet. ;-) http://www.egenix.com/products/python/pyOpenSSL/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 05 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-30: Released eGenix PyRun 1.2.0 ... http://egenix.com/go44 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guy at rzn.co.il Mon May 13 08:37:45 2013 From: guy at rzn.co.il (Guy Rozendorn) Date: Mon, 13 May 2013 09:37:45 +0300 Subject: [Distutils] zc.buildout-2.1.0 is broken on Windows with distribute-0.6.39 Message-ID: distribute-0.6.39 introduces exe wrappers for arm, and there's one cli.exe and gui.exe: 09:30:12 ? guyr-air ? /tmp ? find distribute-0.6.39 -name '*exe' distribute-0.6.39/setuptools/cli-32.exe distribute-0.6.39/setuptools/cli-64.exe distribute-0.6.39/setuptools/cli-arm-32.exe distribute-0.6.39/setuptools/gui-32.exe distribute-0.6.39/setuptools/gui-64.exe distribute-0.6.39/setuptools/gui-arm-32.exe 09:30:15 ? guyr-air ? /tmp ? find distribute-0.6.38 -name '*exe' distribute-0.6.38/setuptools/cli-32.exe distribute-0.6.38/setuptools/cli-64.exe distribute-0.6.38/setuptools/cli.exe distribute-0.6.38/setuptools/gui-32.exe distribute-0.6.38/setuptools/gui-64.exe distribute-0.6.38/setuptools/gui.exe This causes bootstrap.py to fail: Administrator at host-ci46 ~/projector $ python bootstrap.py Creating directory 'C:\\Cygwin\\home\\Administrator\\projector\\bin'. Creating directory 'C:\\Cygwin\\home\\Administrator\\projector\\parts'. Creating directory 'C:\\Cygwin\\home\\Administrator\\projector\\eggs'. Creating directory 'C:\\Cygwin\\home\\Administrator\\projector\\develop-eggs'. While: Bootstrapping. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\buildout.py", line 1923, in main getattr(buildout, command)(args) File "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\buildout.py", line 404, in bootstrap or ''), File "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\easy_install.py", line 960, in scripts initialization, rpsetup) File "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\easy_install.py", line 1064, in _script return _create_script(contents, dest) File "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\easy_install.py", line 1107, in _create_script new_data = pkg_resources.resource_string('setuptools', 'cli.exe') File "C:\Cygwin\home\Administrator\python\lib\site-packages\distribute-0.6.39-py2.7.egg\pkg_resources.py", line 926, in resource_string self, resource_name File "C:\Cygwin\home\Administrator\python\lib\site-packages\distribute-0.6.39-py2.7.egg\pkg_resources.py", line 1199, in get_resource_string return self._get(self._fn(self.module_path, resource_name)) File "C:\Cygwin\home\Administrator\python\lib\site-packages\distribute-0.6.39-py2.7.egg\pkg_resources.py", line 1326, in _get stream = open(path, 'rb') IOError: [Errno 2] No such file or directory: 'C:\\Cygwin\\home\\Administrator\\python\\lib\\site-packages\\distribute-0.6.39-py2.7.egg\\setuptools\\cli.exe' -------------- next part -------------- An HTML attachment was scrubbed... URL: From guy at rzn.co.il Mon May 13 08:44:34 2013 From: guy at rzn.co.il (Guy Rozendorn) Date: Mon, 13 May 2013 09:44:34 +0300 Subject: [Distutils] zc.buildout-2.1.0 is broken on Windows with distribute-0.6.39 In-Reply-To: References: Message-ID: I opened a ticket for distribute to at least leave the old executables in place for easier transition: https://bitbucket.org/tarek/distribute/issue/376/distribute-0639-broke-zcbuildout-210 On Mon, May 13, 2013 at 9:37 AM, Guy Rozendorn wrote: > distribute-0.6.39 introduces exe wrappers for arm, and there's one cli.exe > and gui.exe: > > 09:30:12 ? guyr-air ? /tmp ? find distribute-0.6.39 -name '*exe' > > distribute-0.6.39/setuptools/cli-32.exe > > distribute-0.6.39/setuptools/cli-64.exe > > distribute-0.6.39/setuptools/cli-arm-32.exe > > distribute-0.6.39/setuptools/gui-32.exe > > distribute-0.6.39/setuptools/gui-64.exe > > distribute-0.6.39/setuptools/gui-arm-32.exe > > 09:30:15 ? guyr-air ? /tmp ? find distribute-0.6.38 -name '*exe' > > distribute-0.6.38/setuptools/cli-32.exe > > distribute-0.6.38/setuptools/cli-64.exe > > distribute-0.6.38/setuptools/cli.exe > > distribute-0.6.38/setuptools/gui-32.exe > > distribute-0.6.38/setuptools/gui-64.exe > > distribute-0.6.38/setuptools/gui.exe > > > This causes bootstrap.py to fail: > > Administrator at host-ci46 ~/projector > > $ python bootstrap.py > > Creating directory 'C:\\Cygwin\\home\\Administrator\\projector\\bin'. > > Creating directory 'C:\\Cygwin\\home\\Administrator\\projector\\parts'. > > Creating directory 'C:\\Cygwin\\home\\Administrator\\projector\\eggs'. > > Creating directory > 'C:\\Cygwin\\home\\Administrator\\projector\\develop-eggs'. > > While: > > Bootstrapping. > > > An internal error occured due to a bug in either zc.buildout or in a > > recipe being used: > > Traceback (most recent call last): > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\buildout.py", > line 1923, in main > > getattr(buildout, command)(args) > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\buildout.py", > line 404, in bootstrap > > or ''), > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\easy_install.py", > line 960, in scripts > > initialization, rpsetup) > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\easy_install.py", > line 1064, in _script > > return _create_script(contents, dest) > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\zc.buildout-2.1.0-py2.7.egg\zc\buildout\easy_install.py", > line 1107, in _create_script > > new_data = pkg_resources.resource_string('setuptools', 'cli.exe') > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\distribute-0.6.39-py2.7.egg\pkg_resources.py", > line 926, in resource_string > > self, resource_name > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\distribute-0.6.39-py2.7.egg\pkg_resources.py", > line 1199, in get_resource_string > > return self._get(self._fn(self.module_path, resource_name)) > > File > "C:\Cygwin\home\Administrator\python\lib\site-packages\distribute-0.6.39-py2.7.egg\pkg_resources.py", > line 1326, in _get > > stream = open(path, 'rb') > > IOError: [Errno 2] No such file or directory: > 'C:\\Cygwin\\home\\Administrator\\python\\lib\\site-packages\\distribute-0.6.39-py2.7.egg\\setuptools\\cli.exe' > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sontek at gmail.com Mon May 13 09:54:12 2013 From: sontek at gmail.com (John Anderson) Date: Mon, 13 May 2013 00:54:12 -0700 Subject: [Distutils] Overriding dependency versions Message-ID: If I have a shared dependency between 2 packages but they have pinned different versions of a 3rd package, is there a way to override this? Most of the time when people pin a version they are doing it because they have tested up to that version, but it doesn't always mean they don't work with a later version. For example: P1: sqlalchemy==0.7.6 P2: sqlalchemy P3: sqlalchemy==0.8 If I want to to override this for all packages and tell them its fine to just use 0.8.1 and ignore whatever they were pinned at, can I? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Mon May 13 13:53:16 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 13 May 2013 07:53:16 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Mon, May 13, 2013 at 3:54 AM, John Anderson wrote: > If I have a shared dependency between 2 packages but they have pinned > different versions of a 3rd package, is there a way to override this? > > Most of the time when people pin a version they are doing it because they > have tested up to that version, but it doesn't always mean they don't work > with a later version. > > For example: > > P1: > sqlalchemy==0.7.6 > > P2: > sqlalchemy > > > P3: > sqlalchemy==0.8 > > > > If I want to to override this for all packages and tell them its fine to > just use 0.8.1 and ignore whatever they were pinned at, can I? Gaaa. Libraries shouldn't pin versions (although min and max versions with reasonable ranges is fine). What tool are you using? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From qwcode at gmail.com Mon May 13 15:08:41 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 13 May 2013 06:08:41 -0700 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: > If I have a shared dependency between 2 packages but they have pinned > different versions of a 3rd package, is there a way to override this? > > Most of the time when people pin a version they are doing it because they > have tested up to that version, but it doesn't always mean they don't work > with a later version. > > For example: > > P1: > sqlalchemy==0.7.6 > > P2: > sqlalchemy > > > P3: > sqlalchemy==0.8 > > > > If I want to to override this for all packages and tell them its fine to > just use 0.8.1 and ignore whatever they were pinned at, can I? > > pip requirement files do this. items in the requirement file override found dependency versions. http://www.pip-installer.org/en/latest/cookbook.html#requirements-files Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Mon May 13 15:57:25 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Mon, 13 May 2013 09:57:25 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Mon, May 13, 2013 at 7:53 AM, Jim Fulton wrote: > Gaaa. Libraries shouldn't pin versions (although min and max > versions with reasonable ranges is fine). As a library author, I'm intrigued. Why shouldn't I or others pin versions? Let's provide a situation where it may be necessary (in my opinion): If I release a library dependent upon a particular API in one version of a dependency and before I release my next version I notice plans to break the existing API, why shouldn't I pin the version to protect users (or at least provide a maximum version) from getting horrible exceptions? I have no guarantee I'll be able to update my library based on the new API quickly enough to get a version out concurrent with the new API on that dependency. From dholth at gmail.com Mon May 13 16:03:30 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 13 May 2013 10:03:30 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: It is very difficult to guarantee that your library will never work with any newer version of some other library, but if you can (e.g. python-dateutil, versions >= 2 being exclusively Python 3 compatible), a < or <= qualifier may be appropriate. != is a good one if you know an exact version of some other library is incompatible. == just causes problems because it gets in the way of the integrator who has a lot more information than the library-author-from-some-time-ago. On Mon, May 13, 2013 at 9:57 AM, Ian Cordasco wrote: > On Mon, May 13, 2013 at 7:53 AM, Jim Fulton wrote: >> Gaaa. Libraries shouldn't pin versions (although min and max >> versions with reasonable ranges is fine). > > As a library author, I'm intrigued. Why shouldn't I or others pin > versions? Let's provide a situation where it may be necessary (in my > opinion): > > If I release a library dependent upon a particular API in one version > of a dependency and before I release my next version I notice plans to > break the existing API, why shouldn't I pin the version to protect > users (or at least provide a maximum version) from getting horrible > exceptions? I have no guarantee I'll be able to update my library > based on the new API quickly enough to get a version out concurrent > with the new API on that dependency. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From reinout at vanrees.org Mon May 13 16:12:07 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 13 May 2013 16:12:07 +0200 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On 13-05-13 15:57, Ian Cordasco wrote: > If I release a library dependent upon a particular API in one version > of a dependency and before I release my next version I notice plans to > break the existing API, why shouldn't I pin the version to protect > users (or at least provide a maximum version) from getting horrible > exceptions? Best example: if you pin "somelibrary=1.2" in your library, none of your users can use the critical 1.2.1 bugfix release. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From graffatcolmingov at gmail.com Mon May 13 16:16:38 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Mon, 13 May 2013 10:16:38 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Mon, May 13, 2013 at 10:12 AM, Reinout van Rees wrote: > On 13-05-13 15:57, Ian Cordasco wrote: >> >> If I release a library dependent upon a particular API in one version >> of a dependency and before I release my next version I notice plans to >> break the existing API, why shouldn't I pin the version to protect >> users (or at least provide a maximum version) from getting horrible >> exceptions? > > > Best example: if you pin "somelibrary=1.2" in your library, none of your > users can use the critical 1.2.1 bugfix release. Thanks to you and Daniel. (I accidentally replied to Daniel off-list.) I had just been advised by someone formerly a part of the distutils (or distribute, I forget which) team that you either pin or don't. I try not to, but there have been occasions where I found it necessary. I'll certainly move forward a better developer for your explanations. Thanks again, Ian From dholth at gmail.com Mon May 13 16:20:08 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 13 May 2013 10:20:08 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Mon, May 13, 2013 at 10:03 AM, Daniel Holth wrote: > It is very difficult to guarantee that your library will never work > with any newer version of some other library, but if you can (e.g. > python-dateutil, versions >= 2 being exclusively Python 3 compatible), > a < or <= qualifier may be appropriate. != is a good one if you know > an exact version of some other library is incompatible. == just causes > problems because it gets in the way of the integrator who has a lot > more information than the library-author-from-some-time-ago. The confusion about when to use == comes from the deployment side. It is common to use == in a requirements file when deploying (you tested your application on some exact set of versions), rooting your project's dependency graph with exact distributions. It is almost never useful for interior nodes of a dependency graph (setup.py's install_requires) to declare dependencies on exact versions. a==4 -> b -> c vs. a -> b==7 -> c Usually == crops up when something breaks and the packager decides to pin the known working version of some dependency in setup.py instead of forbidding the known-broken version. From sontek at gmail.com Mon May 13 18:48:14 2013 From: sontek at gmail.com (John Anderson) Date: Mon, 13 May 2013 09:48:14 -0700 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Mon, May 13, 2013 at 6:08 AM, Marcus Smith wrote: > > If I have a shared dependency between 2 packages but they have pinned >> different versions of a 3rd package, is there a way to override this? >> >> Most of the time when people pin a version they are doing it because they >> have tested up to that version, but it doesn't always mean they don't work >> with a later version. >> >> For example: >> >> P1: >> sqlalchemy==0.7.6 >> >> P2: >> sqlalchemy >> >> >> P3: >> sqlalchemy==0.8 >> >> >> >> If I want to to override this for all packages and tell them its fine to >> just use 0.8.1 and ignore whatever they were pinned at, can I? >> >> > pip requirement files do this. > items in the requirement file override found dependency versions. > http://www.pip-installer.org/en/latest/cookbook.html#requirements-files > > Marcus > > We use easy_install/setup.py instead of pip requirements files because there is no way to configure pip to look at a different index via an application configuration file (or at least we couldn't find one). With easy_install we are able to do: [easy_install] index_url = and everything works. With pip it seems to only support a system wide/user wide configuration but not for a specific application/repo. Thanks for the info though! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon May 13 18:57:16 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 13 May 2013 12:57:16 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: You can include many of pip's command line arguments in its requirements files; try putting --index-url right there, seems to work on the tip: --index-url=someindex ... other requirements On Mon, May 13, 2013 at 12:48 PM, John Anderson wrote: > > > > On Mon, May 13, 2013 at 6:08 AM, Marcus Smith wrote: >> >> >>> If I have a shared dependency between 2 packages but they have pinned >>> different versions of a 3rd package, is there a way to override this? >>> >>> Most of the time when people pin a version they are doing it because they >>> have tested up to that version, but it doesn't always mean they don't work >>> with a later version. >>> >>> For example: >>> >>> P1: >>> sqlalchemy==0.7.6 >>> >>> P2: >>> sqlalchemy >>> >>> >>> P3: >>> sqlalchemy==0.8 >>> >>> >>> >>> If I want to to override this for all packages and tell them its fine to >>> just use 0.8.1 and ignore whatever they were pinned at, can I? >>> >> >> pip requirement files do this. >> items in the requirement file override found dependency versions. >> http://www.pip-installer.org/en/latest/cookbook.html#requirements-files >> >> Marcus >> > > We use easy_install/setup.py instead of pip requirements files because there > is no way to configure pip to look at a different index via an application > configuration file (or at least we couldn't find one). > > With easy_install we are able to do: > > [easy_install] > index_url = > > and everything works. With pip it seems to only support a system wide/user > wide configuration but not for a specific application/repo. > > > Thanks for the info though! > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From qwcode at gmail.com Mon May 13 19:59:03 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 13 May 2013 10:59:03 -0700 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: see here in the docs http://www.pip-installer.org/en/latest/logic.html#requirements-file-format On Mon, May 13, 2013 at 9:57 AM, Daniel Holth wrote: > You can include many of pip's command line arguments in its > requirements files; try putting --index-url right there, seems to work > on the tip: > > --index-url=someindex > ... other requirements > > On Mon, May 13, 2013 at 12:48 PM, John Anderson wrote: > > > > > > > > On Mon, May 13, 2013 at 6:08 AM, Marcus Smith wrote: > >> > >> > >>> If I have a shared dependency between 2 packages but they have pinned > >>> different versions of a 3rd package, is there a way to override this? > >>> > >>> Most of the time when people pin a version they are doing it because > they > >>> have tested up to that version, but it doesn't always mean they don't > work > >>> with a later version. > >>> > >>> For example: > >>> > >>> P1: > >>> sqlalchemy==0.7.6 > >>> > >>> P2: > >>> sqlalchemy > >>> > >>> > >>> P3: > >>> sqlalchemy==0.8 > >>> > >>> > >>> > >>> If I want to to override this for all packages and tell them its fine > to > >>> just use 0.8.1 and ignore whatever they were pinned at, can I? > >>> > >> > >> pip requirement files do this. > >> items in the requirement file override found dependency versions. > >> http://www.pip-installer.org/en/latest/cookbook.html#requirements-files > >> > >> Marcus > >> > > > > We use easy_install/setup.py instead of pip requirements files because > there > > is no way to configure pip to look at a different index via an > application > > configuration file (or at least we couldn't find one). > > > > With easy_install we are able to do: > > > > [easy_install] > > index_url = > > > > and everything works. With pip it seems to only support a system > wide/user > > wide configuration but not for a specific application/repo. > > > > > > Thanks for the info though! > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Mon May 13 20:02:45 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 13 May 2013 14:02:45 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Mon, May 13, 2013 at 10:16 AM, Ian Cordasco wrote: > On Mon, May 13, 2013 at 10:12 AM, Reinout van Rees wrote: >> On 13-05-13 15:57, Ian Cordasco wrote: >>> >>> If I release a library dependent upon a particular API in one version >>> of a dependency and before I release my next version I notice plans to >>> break the existing API, why shouldn't I pin the version to protect >>> users (or at least provide a maximum version) from getting horrible >>> exceptions? >> >> >> Best example: if you pin "somelibrary=1.2" in your library, none of your >> users can use the critical 1.2.1 bugfix release. > > Thanks to you and Daniel. (I accidentally replied to Daniel off-list.) > I had just been advised by someone formerly a part of the distutils > (or distribute, I forget which) team that you either pin or don't. I > try not to, but there have been occasions where I found it necessary. > I'll certainly move forward a better developer for your explanations. Unfortunately, when people discuss solutions, they often forget to state the problem they're thinking of. As Daniel pointed out, when deploying an *application*, you should generally pin all of your dependencies, so you can reproduce the deployment. This can be done via the apps setup.cfg, pip requirements or buildout versions. When releasing libraries, you should restrict versions as little as possible. The more you restrict dependency versions, the harder your library is to use. A common recommendation is to set a minimum version to the lowest version of the dependency known to work and to set the maximum version to less then the next major release: >=2, <3dev In practice, most people just set a lower bound, if any. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Mon May 13 21:08:49 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 13 May 2013 15:08:49 -0400 Subject: [Distutils] RFC: Egg cache fro self-contained buildout Message-ID: Problem ======= For (stage and production) deployment purposed, we, ZC, use RPMs. It's considered good hygene to produce source RPMs as well as binary RPMS. This led me to create zc.sourcerelease, which automates creation of self-contained source tar balls, that, among other benefits, provide input for making source RPMs, which feed into a process for creating binary RPMs. We're moving toward a continuous deployment pipeline, where binaries are produced early in the development cycle and tested in a controlled environment that matches production. It no-longer (never did actually) makes sense to produce source RPMs that could be deployed in alternate (untested) environments. In general, our existing build process is grotesquely slow: - we run a buildout to produce a source release. - we run it again to build a source (and binary) rpm from the source release. Both of these run in such a way that all of the eggs have to be rebuilt. (But sources don't have to be downloaded.) We'd like to move toward a model where we construct a build environment for each controlled deployment environment. In this build environment, we never want to build a given distribution more than once. We need to produce application binaries that are self contained. Buildout allows you to use a shared eggs directory. This can greatly speed buildouts, because already-built distributions can be found and used locally. However, buildouts that use shared eggs directories aren't self contained. They depend on the shared eggs directory. I'd like to be able to reuse previously-built eggs, but have eggs installed in my local buildout, so it's self contained. Proposal: egg-cache =================== If egg-cache is set to a directory, then when buildout builds an egg, it will copy it to the egg cache. When looking for distributions, it will look in the egg cache and, if it finds a matching egg there, it will copy the egg to the buildout eggs directory. The end result will be that an egg cache will have the same economy as the current shared eggs directory, as far as building is concerned, but it won't have the disk-space saving of a shared eggs directiory. It will lead to buildouts that are self contained (at least wrt eggs) and that can be copied to a deployment environment directly. Thoughts? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From dholth at gmail.com Mon May 13 21:26:37 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 13 May 2013 15:26:37 -0400 Subject: [Distutils] RFC: Egg cache fro self-contained buildout In-Reply-To: References: Message-ID: Sounds like a nice feature. I like having copies of everything (and nothing extra) for each deployment. On Mon, May 13, 2013 at 3:08 PM, Jim Fulton wrote: > Problem > ======= > > For (stage and production) deployment purposed, we, ZC, use RPMs. > It's considered good hygene to produce source RPMs as well as binary > RPMS. This led me to create zc.sourcerelease, which automates > creation of self-contained source tar balls, that, among other > benefits, provide input for making source RPMs, which feed into a > process for creating binary RPMs. > > We're moving toward a continuous deployment pipeline, where binaries > are produced early in the development cycle and tested in a controlled > environment that matches production. It no-longer (never did actually) > makes sense to produce source RPMs that could be deployed in alternate > (untested) environments. > > In general, our existing build process is grotesquely slow: > > - we run a buildout to produce a source release. > > - we run it again to build a source (and binary) rpm from the source > release. > > Both of these run in such a way that all of the eggs have to be > rebuilt. (But sources don't have to be downloaded.) > > We'd like to move toward a model where we construct a build > environment for each controlled deployment environment. In this build > environment, we never want to build a given distribution more than > once. We need to produce application binaries that are self > contained. > > Buildout allows you to use a shared eggs directory. This can greatly > speed buildouts, because already-built distributions can be found and > used locally. However, buildouts that use shared eggs directories > aren't self contained. They depend on the shared eggs directory. > I'd like to be able to reuse previously-built eggs, but have eggs > installed in my local buildout, so it's self contained. > > Proposal: egg-cache > =================== > > If egg-cache is set to a directory, then when buildout builds an egg, > it will copy it to the egg cache. When looking for distributions, it > will look in the egg cache and, if it finds a matching egg there, it > will copy the egg to the buildout eggs directory. > > The end result will be that an egg cache will have the same economy as > the current shared eggs directory, as far as building is concerned, > but it won't have the disk-space saving of a shared eggs directiory. > It will lead to buildouts that are self contained (at least wrt eggs) > and that can be copied to a deployment environment directly. > > Thoughts? > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From leorochael at gmail.com Mon May 13 22:07:42 2013 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Mon, 13 May 2013 17:07:42 -0300 Subject: [Distutils] RFC: Egg cache fro self-contained buildout In-Reply-To: References: Message-ID: +1 This is potentially useful for Heroku or Google Apps, or anything that requires all your files to exist inside the build area after build time from where it creates the blob that is actually pushed to all the slaves. On Mon, May 13, 2013 at 4:26 PM, Daniel Holth wrote: > Sounds like a nice feature. I like having copies of everything (and > nothing extra) for each deployment. > > On Mon, May 13, 2013 at 3:08 PM, Jim Fulton wrote: >> Problem >> ======= >> >> For (stage and production) deployment purposed, we, ZC, use RPMs. >> It's considered good hygene to produce source RPMs as well as binary >> RPMS. This led me to create zc.sourcerelease, which automates >> creation of self-contained source tar balls, that, among other >> benefits, provide input for making source RPMs, which feed into a >> process for creating binary RPMs. >> >> We're moving toward a continuous deployment pipeline, where binaries >> are produced early in the development cycle and tested in a controlled >> environment that matches production. It no-longer (never did actually) >> makes sense to produce source RPMs that could be deployed in alternate >> (untested) environments. >> >> In general, our existing build process is grotesquely slow: >> >> - we run a buildout to produce a source release. >> >> - we run it again to build a source (and binary) rpm from the source >> release. >> >> Both of these run in such a way that all of the eggs have to be >> rebuilt. (But sources don't have to be downloaded.) >> >> We'd like to move toward a model where we construct a build >> environment for each controlled deployment environment. In this build >> environment, we never want to build a given distribution more than >> once. We need to produce application binaries that are self >> contained. >> >> Buildout allows you to use a shared eggs directory. This can greatly >> speed buildouts, because already-built distributions can be found and >> used locally. However, buildouts that use shared eggs directories >> aren't self contained. They depend on the shared eggs directory. >> I'd like to be able to reuse previously-built eggs, but have eggs >> installed in my local buildout, so it's self contained. >> >> Proposal: egg-cache >> =================== >> >> If egg-cache is set to a directory, then when buildout builds an egg, >> it will copy it to the egg cache. When looking for distributions, it >> will look in the egg cache and, if it finds a matching egg there, it >> will copy the egg to the buildout eggs directory. >> >> The end result will be that an egg cache will have the same economy as >> the current shared eggs directory, as far as building is concerned, >> but it won't have the disk-space saving of a shared eggs directiory. >> It will lead to buildouts that are self contained (at least wrt eggs) >> and that can be copied to a deployment environment directly. >> >> Thoughts? >> >> Jim >> >> -- >> Jim Fulton >> http://www.linkedin.com/in/jimfulton >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From kamal.mustafa at gmail.com Tue May 14 04:27:18 2013 From: kamal.mustafa at gmail.com (Mohd Kamal Bin Mustafa) Date: Tue, 14 May 2013 10:27:18 +0800 Subject: [Distutils] RFC: Egg cache fro self-contained buildout In-Reply-To: References: Message-ID: +1 For deployment, I have a script that create a svn tag from trunk, run buildout and create tarball of the resulting buildout. On the production, another script will unpack the tarball and rerun buildout but this time with -N -o and since the eggs already in the tarball, it's very quick and it just regenerate the script and config file to reflect the actual production environment. The problem was when creating the release, as it's a fresh checkout from svn, the eggs dir not populated yet and due to issue [1], the buildout was very slow. While I can do something like copy the eggs dir from my current working directory into the tag dir before running buildout, it will make the eggs not "pristine" anymore since I might have tampered it with debugging stuff, it's in my working directory after all. [1]:https://github.com/buildout/buildout/issues/116 On Tue, May 14, 2013 at 3:08 AM, Jim Fulton wrote: > Problem > ======= > > For (stage and production) deployment purposed, we, ZC, use RPMs. > It's considered good hygene to produce source RPMs as well as binary > RPMS. This led me to create zc.sourcerelease, which automates > creation of self-contained source tar balls, that, among other > benefits, provide input for making source RPMs, which feed into a > process for creating binary RPMs. > > We're moving toward a continuous deployment pipeline, where binaries > are produced early in the development cycle and tested in a controlled > environment that matches production. It no-longer (never did actually) > makes sense to produce source RPMs that could be deployed in alternate > (untested) environments. > > In general, our existing build process is grotesquely slow: > > - we run a buildout to produce a source release. > > - we run it again to build a source (and binary) rpm from the source > release. > > Both of these run in such a way that all of the eggs have to be > rebuilt. (But sources don't have to be downloaded.) > > We'd like to move toward a model where we construct a build > environment for each controlled deployment environment. In this build > environment, we never want to build a given distribution more than > once. We need to produce application binaries that are self > contained. > > Buildout allows you to use a shared eggs directory. This can greatly > speed buildouts, because already-built distributions can be found and > used locally. However, buildouts that use shared eggs directories > aren't self contained. They depend on the shared eggs directory. > I'd like to be able to reuse previously-built eggs, but have eggs > installed in my local buildout, so it's self contained. > > Proposal: egg-cache > =================== > > If egg-cache is set to a directory, then when buildout builds an egg, > it will copy it to the egg cache. When looking for distributions, it > will look in the egg cache and, if it finds a matching egg there, it > will copy the egg to the buildout eggs directory. > > The end result will be that an egg cache will have the same economy as > the current shared eggs directory, as far as building is concerned, > but it won't have the disk-space saving of a shared eggs directiory. > It will lead to buildouts that are self contained (at least wrt eggs) > and that can be copied to a deployment environment directly. > > Thoughts? > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From dw at botanicus.net Wed May 15 00:25:33 2013 From: dw at botanicus.net (David Wilson) Date: Tue, 14 May 2013 23:25:33 +0100 Subject: [Distutils] Fixing Cheese Shop search In-Reply-To: References: Message-ID: Quasi-monthly ping :) I'm still happy to volunteer help. David On 23 April 2013 13:41, Nick Coghlan wrote: > On Mon, Apr 22, 2013 at 11:30 PM, David Wilson wrote: >>> Prototype code is here: https://bitbucket.org/dmw/pypi-search (note: >>> relies on a pre-alpha quality DB library I'm hacking on) >>> >>> Thoughts? > > Hi David, > > This certainly sounds intriguing (and even promising), but I believe > Richard (Jones, the main PyPI maintainer) has just returned from a > long business trip, so it may be a while before he catches back up > with distutils-sig. > > I'll see if I can nudge some other folks and get you a better answer... > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Wed May 15 04:36:59 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 14 May 2013 22:36:59 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: Why don't we simply provide an option to ignore == if it is not a "root" dependency? On Mon, May 13, 2013 at 2:02 PM, Jim Fulton wrote: > On Mon, May 13, 2013 at 10:16 AM, Ian Cordasco > wrote: >> On Mon, May 13, 2013 at 10:12 AM, Reinout van Rees wrote: >>> On 13-05-13 15:57, Ian Cordasco wrote: >>>> >>>> If I release a library dependent upon a particular API in one version >>>> of a dependency and before I release my next version I notice plans to >>>> break the existing API, why shouldn't I pin the version to protect >>>> users (or at least provide a maximum version) from getting horrible >>>> exceptions? >>> >>> >>> Best example: if you pin "somelibrary=1.2" in your library, none of your >>> users can use the critical 1.2.1 bugfix release. >> >> Thanks to you and Daniel. (I accidentally replied to Daniel off-list.) >> I had just been advised by someone formerly a part of the distutils >> (or distribute, I forget which) team that you either pin or don't. I >> try not to, but there have been occasions where I found it necessary. >> I'll certainly move forward a better developer for your explanations. > > Unfortunately, when people discuss solutions, they often > forget to state the problem they're thinking of. > > As Daniel pointed out, when deploying an *application*, you should > generally pin all of your dependencies, so you can reproduce the > deployment. This can be done via the apps setup.cfg, pip requirements > or buildout versions. > > When releasing libraries, you should restrict versions as little as > possible. The more you restrict dependency versions, the harder your > library is to use. A common recommendation is to set a minimum version > to the lowest version of the dependency known to work and to set the > maximum version to less then the next major release: > > >=2, <3dev > > In practice, most people just set a lower bound, if any. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Wed May 15 05:44:55 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 14 May 2013 23:44:55 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name Message-ID: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> Currently PyPI allows a project name to contain basically any character except for a /. However most of the installation tooling doesn't not work with this wide of a namespace. It also opens up several avenues for spoofing attack where you trick people into copy and pasting an install command that looks like you're installing one package but you are really installing a different one. So I propose that moving forward that all projects/distributions are required to have names using only urlsafe characters. Specifically letters, decimal digits, hyphen, period, and underscore. Doing this would allow a better experience for people attempting to install packages, it would allow tool authors to test and make sure they can install all valid packages etc. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed May 15 05:56:13 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 14 May 2013 23:56:13 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> Message-ID: Surely it has to be Unicode. Why not reuse the python 3 identifier rules, or just Unicode alphanumeric and underscore. Will miss the snowman. On May 14, 2013 11:45 PM, "Donald Stufft" wrote: > Currently PyPI allows a project name to contain basically any character > except for a /. However most of the installation tooling doesn't not work > with this wide of a namespace. It also opens up several avenues for > spoofing attack where you trick people into copy and pasting an install > command that looks like you're installing one package but you are really > installing a different one. > > So I propose that moving forward that all projects/distributions are > required to have names using only urlsafe characters. Specifically letters, > decimal digits, hyphen, period, and underscore. > > Doing this would allow a better experience for people attempting to > install packages, it would allow tool authors to test and make sure they > can install all valid packages etc. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed May 15 05:58:25 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 14 May 2013 23:58:25 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> Message-ID: <890CB87F-F41A-48E7-BA50-D0A6FDAE2026@stufft.io> On May 14, 2013, at 11:56 PM, Daniel Holth wrote: > Surely it has to be Unicode. Why not reuse the python 3 identifier rules, or just Unicode alphanumeric and underscore. Will miss the snowman. > > On May 14, 2013 11:45 PM, "Donald Stufft" wrote: > Currently PyPI allows a project name to contain basically any character except for a /. However most of the installation tooling doesn't not work with this wide of a namespace. It also opens up several avenues for spoofing attack where you trick people into copy and pasting an install command that looks like you're installing one package but you are really installing a different one. > > So I propose that moving forward that all projects/distributions are required to have names using only urlsafe characters. Specifically letters, decimal digits, hyphen, period, and underscore. > > Doing this would allow a better experience for people attempting to install packages, it would allow tool authors to test and make sure they can install all valid packages etc. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > Allowing unicode means you can do things like find glyphs that are technically different but look the same to most people. This isn't really a problem in code you're writing but it could be a problem for malicious tutorials and such. It's also a problem because currently "Django>=1.5" is a valid identifier for a package, but tools have no way to know if I'm asking them to install Django version 1.5 or greater, or if I want them to install "Django>=1.5" any version. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 15 06:00:32 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 00:00:32 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <890CB87F-F41A-48E7-BA50-D0A6FDAE2026@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <890CB87F-F41A-48E7-BA50-D0A6FDAE2026@stufft.io> Message-ID: <5C76A398-A954-4F04-B3FB-214D24752E8F@stufft.io> On May 14, 2013, at 11:58 PM, Donald Stufft wrote: > On May 14, 2013, at 11:56 PM, Daniel Holth wrote: > >> Surely it has to be Unicode. Why not reuse the python 3 identifier rules, or just Unicode alphanumeric and underscore. Will miss the snowman. >> >> On May 14, 2013 11:45 PM, "Donald Stufft" wrote: >> Currently PyPI allows a project name to contain basically any character except for a /. However most of the installation tooling doesn't not work with this wide of a namespace. It also opens up several avenues for spoofing attack where you trick people into copy and pasting an install command that looks like you're installing one package but you are really installing a different one. >> >> So I propose that moving forward that all projects/distributions are required to have names using only urlsafe characters. Specifically letters, decimal digits, hyphen, period, and underscore. >> >> Doing this would allow a better experience for people attempting to install packages, it would allow tool authors to test and make sure they can install all valid packages etc. >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> > > > Allowing unicode means you can do things like find glyphs that are technically different but look the same to most people. This isn't really a problem in code you're writing but it could be a problem for malicious tutorials and such. > > It's also a problem because currently "Django>=1.5" is a valid identifier for a package, but tools have no way to know if I'm asking them to install Django version 1.5 or greater, or if I want them to install "Django>=1.5" any version. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > Also python identifier are typically confined only within the python code files and runtimes, however there's no promise that all the places people need to use these values will be unicode aware. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 15 06:11:07 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 00:11:07 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> Message-ID: <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> On May 14, 2013, at 11:44 PM, Donald Stufft wrote: > Currently PyPI allows a project name to contain basically any character except for a /. However most of the installation tooling doesn't not work with this wide of a namespace. It also opens up several avenues for spoofing attack where you trick people into copy and pasting an install command that looks like you're installing one package but you are really installing a different one. > > So I propose that moving forward that all projects/distributions are required to have names using only urlsafe characters. Specifically letters, decimal digits, hyphen, period, and underscore. > > Doing this would allow a better experience for people attempting to install packages, it would allow tool authors to test and make sure they can install all valid packages etc. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig For more information data using a few days old copy of the database on Crate: SELECT COUNT(*) FROM packages WHERE name ~* '^[-a-z0-9_\.]+$'; count ------- 30422 SELECT COUNT(*) FROM packages WHERE name !~* '^[-a-z0-9_\.]+$'; count ------- 225 So this would disallow 225 (0.7%) and would not affect 30422 (99.3%) total names that are currently in use on PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed May 15 06:36:16 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 00:36:16 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> Message-ID: >= would certainty not be a valid name. So I agree with you about restrictions except possibly on the set of allowed characters. Of course the weird names aren't on pypi yet, the current tooling has bad Unicode support. Pep 3131 pretty much sums up this issue and the objections exactly, if you search/replace. It begins: Python code is written by many people in the world who are not familiar with the English language, or even well-acquainted with the Latin writing system. Such developers often desire to define classes and functions with names in their native languages, rather than having to come up with an (often incorrect) English translation of the concept they want to name. By using identifiers in their native language, code clarity and maintainability of the code among speakers of that language improves. On May 15, 2013 12:11 AM, "Donald Stufft" wrote: > > On May 14, 2013, at 11:44 PM, Donald Stufft wrote: > > > Currently PyPI allows a project name to contain basically any character > except for a /. However most of the installation tooling doesn't not work > with this wide of a namespace. It also opens up several avenues for > spoofing attack where you trick people into copy and pasting an install > command that looks like you're installing one package but you are really > installing a different one. > > > > So I propose that moving forward that all projects/distributions are > required to have names using only urlsafe characters. Specifically letters, > decimal digits, hyphen, period, and underscore. > > > > Doing this would allow a better experience for people attempting to > install packages, it would allow tool authors to test and make sure they > can install all valid packages etc. > > > > ----------------- > > Donald Stufft > > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > For more information data using a few days old copy of the database on > Crate: > > SELECT COUNT(*) FROM packages WHERE name ~* '^[-a-z0-9_\.]+$'; > count > ------- > 30422 > > SELECT COUNT(*) FROM packages WHERE name !~* '^[-a-z0-9_\.]+$'; > count > ------- > 225 > > So this would disallow 225 (0.7%) and would not affect 30422 (99.3%) total > names that are currently in use on PyPI. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed May 15 06:45:35 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 00:45:35 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> Message-ID: <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> On May 15, 2013, at 12:36 AM, Daniel Holth wrote: > >= would certainty not be a valid name. So I agree with you about restrictions except possibly on the set of allowed characters. > > Of course the weird names aren't on pypi yet, the current tooling has bad Unicode support. > > Pep 3131 pretty much sums up this issue and the objections exactly, if you search/replace. It begins: > > Python code is written by many people in the world who are not familiar with the English language, or even well-acquainted with the Latin writing system. Such developers often desire to define classes and functions with names in their native languages, rather than having to come up with an (often incorrect) English translation of the concept they want to name. By using identifiers in their native language, code clarity and maintainability of the code among speakers of that language improves. > The contexts are different. It's unlikely that someone in the same codebase is going to attempt to trick you into running function named f?? instead of foo (those are different by the way). However it is a very simple attack to tell newcomers to ``pip install Djang?`` instead of ``pip install Django`` (again different). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 15 06:54:10 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 00:54:10 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> Message-ID: On May 15, 2013, at 12:45 AM, Donald Stufft wrote: > > On May 15, 2013, at 12:36 AM, Daniel Holth wrote: > >> >= would certainty not be a valid name. So I agree with you about restrictions except possibly on the set of allowed characters. >> >> Of course the weird names aren't on pypi yet, the current tooling has bad Unicode support. >> >> Pep 3131 pretty much sums up this issue and the objections exactly, if you search/replace. It begins: >> >> Python code is written by many people in the world who are not familiar with the English language, or even well-acquainted with the Latin writing system. Such developers often desire to define classes and functions with names in their native languages, rather than having to come up with an (often incorrect) English translation of the concept they want to name. By using identifiers in their native language, code clarity and maintainability of the code among speakers of that language improves. >> > The contexts are different. It's unlikely that someone in the same codebase is going to attempt to trick you into running function named f?? instead of foo (those are different by the way). However it is a very simple attack to tell newcomers to ``pip install Djang?`` instead of ``pip install Django`` (again different). > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Perhaps this better explains my point: http://d.stufft.io/image/2t021y342a1d ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 15 07:03:09 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 01:03:09 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> Message-ID: <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> On May 15, 2013, at 12:54 AM, Donald Stufft wrote: > > On May 15, 2013, at 12:45 AM, Donald Stufft wrote: > >> >> On May 15, 2013, at 12:36 AM, Daniel Holth wrote: >> >>> >= would certainty not be a valid name. So I agree with you about restrictions except possibly on the set of allowed characters. >>> >>> Of course the weird names aren't on pypi yet, the current tooling has bad Unicode support. >>> >>> Pep 3131 pretty much sums up this issue and the objections exactly, if you search/replace. It begins: >>> >>> Python code is written by many people in the world who are not familiar with the English language, or even well-acquainted with the Latin writing system. Such developers often desire to define classes and functions with names in their native languages, rather than having to come up with an (often incorrect) English translation of the concept they want to name. By using identifiers in their native language, code clarity and maintainability of the code among speakers of that language improves. >>> >> The contexts are different. It's unlikely that someone in the same codebase is going to attempt to trick you into running function named f?? instead of foo (those are different by the way). However it is a very simple attack to tell newcomers to ``pip install Djang?`` instead of ``pip install Django`` (again different). >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > Perhaps this better explains my point: http://d.stufft.io/image/2t021y342a1d > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig And an install log, just to prove it's possible: https://gist.github.com/dstufft/5581735 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Wed May 15 07:30:41 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 14 May 2013 22:30:41 -0700 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> Message-ID: <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> On May 14, 2013, at 10:03 PM, Donald Stufft wrote: > > On May 15, 2013, at 12:54 AM, Donald Stufft wrote: > >> >> On May 15, 2013, at 12:45 AM, Donald Stufft wrote: >> >>> >>> On May 15, 2013, at 12:36 AM, Daniel Holth wrote: >>> >>>> >= would certainty not be a valid name. So I agree with you about restrictions except possibly on the set of allowed characters. >>>> >>>> Of course the weird names aren't on pypi yet, the current tooling has bad Unicode support. >>>> >>>> Pep 3131 pretty much sums up this issue and the objections exactly, if you search/replace. It begins: >>>> >>>> Python code is written by many people in the world who are not familiar with the English language, or even well-acquainted with the Latin writing system. Such developers often desire to define classes and functions with names in their native languages, rather than having to come up with an (often incorrect) English translation of the concept they want to name. By using identifiers in their native language, code clarity and maintainability of the code among speakers of that language improves. >>>> >>> The contexts are different. It's unlikely that someone in the same codebase is going to attempt to trick you into running function named f?? instead of foo (those are different by the way). However it is a very simple attack to tell newcomers to ``pip install Djang?`` instead of ``pip install Django`` (again different). >>> >>> ----------------- >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig >> >> Perhaps this better explains my point: http://d.stufft.io/image/2t021y342a1d >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > And an install log, just to prove it's possible: https://gist.github.com/dstufft/5581735 File me as a +1 for this change. If we absolutely must support unicode package names, we should do the URLs in PyPI in punycode and have pip show a puny-mangled name in a confirmation prompt for anything with non-ascii characters in it. Yes, that does basically remove all reason to use unicode in package names, which is why I think blocking it is a much better idea. [a-zA-Z0-9_.-] is probably the right way to go. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed May 15 08:58:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 May 2013 16:58:09 +1000 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> Message-ID: On Wed, May 15, 2013 at 3:30 PM, Noah Kantrowitz wrote: > File me as a +1 for this change. If we absolutely must support unicode package names, we should do the URLs in PyPI in punycode and have pip show a puny-mangled name in a confirmation prompt for anything with non-ascii characters in it. Yes, that does basically remove all reason to use unicode in package names, which is why I think blocking it is a much better idea. [a-zA-Z0-9_.-] is probably the right way to go. Right, I'm also a fan of tightening up the rules for metadata 2.0 and PyPI in general. Fedora's package naming policy is limited to the characters Noah suggests, with "+" also allowed: https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Common_Character_Set_for_Package_Naming And Debian is also similar, with "+" allowed and "_" excluded: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source Given the much higher security risks for distribution commands (over identifiers in code), I think the conservative approach of following Fedora & Debian's example is the right way to go here. Anyone want to run a scan over the PyPI package set to see how many packages would cause problems for a "[a-zA-Z0-9_.-]" only filter? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed May 15 09:29:22 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 03:29:22 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> Message-ID: <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> On May 15, 2013, at 2:58 AM, Nick Coghlan wrote: > On Wed, May 15, 2013 at 3:30 PM, Noah Kantrowitz wrote: >> File me as a +1 for this change. If we absolutely must support unicode package names, we should do the URLs in PyPI in punycode and have pip show a puny-mangled name in a confirmation prompt for anything with non-ascii characters in it. Yes, that does basically remove all reason to use unicode in package names, which is why I think blocking it is a much better idea. [a-zA-Z0-9_.-] is probably the right way to go. > > Right, I'm also a fan of tightening up the rules for metadata 2.0 and > PyPI in general. > > Fedora's package naming policy is limited to the characters Noah > suggests, with "+" also allowed: > https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Common_Character_Set_for_Package_Naming > > And Debian is also similar, with "+" allowed and "_" excluded: > http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source > > Given the much higher security risks for distribution commands (over > identifiers in code), I think the conservative approach of following > Fedora & Debian's example is the right way to go here. > > Anyone want to run a scan over the PyPI package set to see how many > packages would cause problems for a "[a-zA-Z0-9_.-]" only filter? See my previous email where I did queries against my local DB. It's 225 total projects that wouldn't be allowed. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 15 09:31:19 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 03:31:19 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> Message-ID: <7E38541B-8491-4DA6-8556-AAE19BA1F40F@stufft.io> On May 15, 2013, at 2:58 AM, Nick Coghlan wrote: > On Wed, May 15, 2013 at 3:30 PM, Noah Kantrowitz wrote: >> File me as a +1 for this change. If we absolutely must support unicode package names, we should do the URLs in PyPI in punycode and have pip show a puny-mangled name in a confirmation prompt for anything with non-ascii characters in it. Yes, that does basically remove all reason to use unicode in package names, which is why I think blocking it is a much better idea. [a-zA-Z0-9_.-] is probably the right way to go. > > Right, I'm also a fan of tightening up the rules for metadata 2.0 and > PyPI in general. > > Fedora's package naming policy is limited to the characters Noah > suggests, with "+" also allowed: > https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Common_Character_Set_for_Package_Naming > > And Debian is also similar, with "+" allowed and "_" excluded: > http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source > > Given the much higher security risks for distribution commands (over > identifiers in code), I think the conservative approach of following > Fedora & Debian's example is the right way to go here. > > Anyone want to run a scan over the PyPI package set to see how many > packages would cause problems for a "[a-zA-Z0-9_.-]" only filter? > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia Excluding _ might be a good idea as well because of how easy it is to mistake it for - I hadn't thought of that. Currently PyPI guarantees uniqueness using only alpha numerics and the "-" character so it wouldn't be hard to do this but isn't strictly required as PyPI won't allow foo-bar and foo_bar. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From richard at python.org Wed May 15 11:47:15 2013 From: richard at python.org (Richard Jones) Date: Wed, 15 May 2013 19:47:15 +1000 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <7E38541B-8491-4DA6-8556-AAE19BA1F40F@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <7E38541B-8491-4DA6-8556-AAE19BA1F40F@stufft.io> Message-ID: +1 for this change, even though it's more heat for me ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed May 15 13:10:00 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 07:10:00 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> Message-ID: <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> On May 15, 2013, at 6:21 AM, "Eric V. Smith" wrote: > On May 15, 2013, at 3:29 AM, Donald Stufft wrote: > >> >> On May 15, 2013, at 2:58 AM, Nick Coghlan wrote: >> >>> On Wed, May 15, 2013 at 3:30 PM, Noah Kantrowitz wrote: >>>> File me as a +1 for this change. If we absolutely must support unicode package names, we should do the URLs in PyPI in punycode and have pip show a puny-mangled name in a confirmation prompt for anything with non-ascii characters in it. Yes, that does basically remove all reason to use unicode in package names, which is why I think blocking it is a much better idea. [a-zA-Z0-9_.-] is probably the right way to go. >>> >>> Right, I'm also a fan of tightening up the rules for metadata 2.0 and >>> PyPI in general. >>> >>> Fedora's package naming policy is limited to the characters Noah >>> suggests, with "+" also allowed: >>> https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Common_Character_Set_for_Package_Naming >>> >>> And Debian is also similar, with "+" allowed and "_" excluded: >>> http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source >>> >>> Given the much higher security risks for distribution commands (over >>> identifiers in code), I think the conservative approach of following >>> Fedora & Debian's example is the right way to go here. >>> >>> Anyone want to run a scan over the PyPI package set to see how many >>> packages would cause problems for a "[a-zA-Z0-9_.-]" only filter? >> >> See my previous email where I did queries against my local DB. It's 225 total projects that wouldn't be allowed. > > Can you send the list of those projects? > > Eric. > Here you go https://gist.github.com/dstufft/5583225 used a Python oneliner and the PyPI API so others can reproduce easily if they wish. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eric at trueblade.com Wed May 15 13:17:40 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 15 May 2013 07:17:40 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> Message-ID: <51936ED4.2090901@trueblade.com> On 05/15/2013 07:10 AM, Donald Stufft wrote: >>>> Anyone want to run a scan over the PyPI package set to see >>>> how many packages would cause problems for a "[a-zA-Z0-9_.-]" >>>> only filter? >>> >>> See my previous email where I did queries against my local DB. >>> It's 225 total projects that wouldn't be allowed. >> >> Can you send the list of those projects? >> >> Eric. >> > > Here you go https://gist.github.com/dstufft/5583225 used a Python > oneliner and the PyPI API so others can reproduce easily if they > wish. Perfect. Thanks. It looks like space causes most of the issues. I'm not sure how "Twisted Flow >= 1.0" would be expected to parse. Eric. From dholth at gmail.com Wed May 15 13:12:12 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 07:12:12 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <7E38541B-8491-4DA6-8556-AAE19BA1F40F@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <7E38541B-8491-4DA6-8556-AAE19BA1F40F@stufft.io> Message-ID: If it's a security problem, why not do confusability detection on the server instead? - and _ are essentially the same character in setuptools. On Wed, May 15, 2013 at 3:31 AM, Donald Stufft wrote: > > On May 15, 2013, at 2:58 AM, Nick Coghlan wrote: > >> On Wed, May 15, 2013 at 3:30 PM, Noah Kantrowitz wrote: >>> File me as a +1 for this change. If we absolutely must support unicode package names, we should do the URLs in PyPI in punycode and have pip show a puny-mangled name in a confirmation prompt for anything with non-ascii characters in it. Yes, that does basically remove all reason to use unicode in package names, which is why I think blocking it is a much better idea. [a-zA-Z0-9_.-] is probably the right way to go. >> >> Right, I'm also a fan of tightening up the rules for metadata 2.0 and >> PyPI in general. >> >> Fedora's package naming policy is limited to the characters Noah >> suggests, with "+" also allowed: >> https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Common_Character_Set_for_Package_Naming >> >> And Debian is also similar, with "+" allowed and "_" excluded: >> http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source >> >> Given the much higher security risks for distribution commands (over >> identifiers in code), I think the conservative approach of following >> Fedora & Debian's example is the right way to go here. >> >> Anyone want to run a scan over the PyPI package set to see how many >> packages would cause problems for a "[a-zA-Z0-9_.-]" only filter? >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > Excluding _ might be a good idea as well because of how easy it is to mistake it for - I hadn't thought of that. Currently PyPI guarantees uniqueness using only alpha numerics and the "-" character so it wouldn't be hard to do this but isn't strictly required as PyPI won't allow foo-bar and foo_bar. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From eric at trueblade.com Wed May 15 12:21:27 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 15 May 2013 06:21:27 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> Message-ID: <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> On May 15, 2013, at 3:29 AM, Donald Stufft wrote: > > On May 15, 2013, at 2:58 AM, Nick Coghlan wrote: > >> On Wed, May 15, 2013 at 3:30 PM, Noah Kantrowitz wrote: >>> File me as a +1 for this change. If we absolutely must support unicode package names, we should do the URLs in PyPI in punycode and have pip show a puny-mangled name in a confirmation prompt for anything with non-ascii characters in it. Yes, that does basically remove all reason to use unicode in package names, which is why I think blocking it is a much better idea. [a-zA-Z0-9_.-] is probably the right way to go. >> >> Right, I'm also a fan of tightening up the rules for metadata 2.0 and >> PyPI in general. >> >> Fedora's package naming policy is limited to the characters Noah >> suggests, with "+" also allowed: >> https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Common_Character_Set_for_Package_Naming >> >> And Debian is also similar, with "+" allowed and "_" excluded: >> http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source >> >> Given the much higher security risks for distribution commands (over >> identifiers in code), I think the conservative approach of following >> Fedora & Debian's example is the right way to go here. >> >> Anyone want to run a scan over the PyPI package set to see how many >> packages would cause problems for a "[a-zA-Z0-9_.-]" only filter? > > See my previous email where I did queries against my local DB. It's 225 total projects that wouldn't be allowed. Can you send the list of those projects? Eric. From dholth at gmail.com Wed May 15 16:31:56 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 10:31:56 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <51936ED4.2090901@trueblade.com> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> Message-ID: How to avoid confusables. These scripts are recommended for use in identifiers: http://www.unicode.org/reports/tr31/#Table_Recommended_Scripts This report details a confusables detection algorithm: http://www.unicode.org/reports/tr39/#Confusable_Detection And ICU implements it: http://www.icu-project.org/apiref/icu4c/uspoof_8h.html (see also PyICU). The package index would enforce uniqueness of the "skeleton" of each registered package which is just an internal normalization based on confusability. if skeleton(identifier1) == skeleton(identifier2) then id1 and id2 are confusable. The tooling could get away with a simpler rule like re.sub("[^\w\d.]+", "_", distribution, re.UNICODE) As a bonus to including the world, this should be able to prevent people from exchanging zeroes for capital O. On Wed, May 15, 2013 at 7:17 AM, Eric V. Smith wrote: > On 05/15/2013 07:10 AM, Donald Stufft wrote: >>>>> Anyone want to run a scan over the PyPI package set to see >>>>> how many packages would cause problems for a "[a-zA-Z0-9_.-]" >>>>> only filter? >>>> >>>> See my previous email where I did queries against my local DB. >>>> It's 225 total projects that wouldn't be allowed. >>> >>> Can you send the list of those projects? >>> >>> Eric. >>> >> >> Here you go https://gist.github.com/dstufft/5583225 used a Python >> oneliner and the PyPI API so others can reproduce easily if they >> wish. > > Perfect. Thanks. > > It looks like space causes most of the issues. I'm not sure how > "Twisted Flow >= 1.0" would be expected to parse. > > Eric. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Wed May 15 19:07:30 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 13:07:30 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> Message-ID: <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> On May 15, 2013, at 10:31 AM, Daniel Holth wrote: > How to avoid confusables. > > These scripts are recommended for use in identifiers: > http://www.unicode.org/reports/tr31/#Table_Recommended_Scripts > > This report details a confusables detection algorithm: > http://www.unicode.org/reports/tr39/#Confusable_Detection > > And ICU implements it: > http://www.icu-project.org/apiref/icu4c/uspoof_8h.html (see also > PyICU). > > The package index would enforce uniqueness of the "skeleton" of each > registered package which is just an internal normalization based on > confusability. if skeleton(identifier1) == skeleton(identifier2) then > id1 and id2 are confusable. > > The tooling could get away with a simpler rule like > re.sub("[^\w\d.]+", "_", distribution, re.UNICODE) > > As a bonus to including the world, this should be able to prevent > people from exchanging zeroes for capital O. > > On Wed, May 15, 2013 at 7:17 AM, Eric V. Smith wrote: >> On 05/15/2013 07:10 AM, Donald Stufft wrote: >>>>>> Anyone want to run a scan over the PyPI package set to see >>>>>> how many packages would cause problems for a "[a-zA-Z0-9_.-]" >>>>>> only filter? >>>>> >>>>> See my previous email where I did queries against my local DB. >>>>> It's 225 total projects that wouldn't be allowed. >>>> >>>> Can you send the list of those projects? >>>> >>>> Eric. >>>> >>> >>> Here you go https://gist.github.com/dstufft/5583225 used a Python >>> oneliner and the PyPI API so others can reproduce easily if they >>> wish. >> >> Perfect. Thanks. >> >> It looks like space causes most of the issues. I'm not sure how >> "Twisted Flow >= 1.0" would be expected to parse. >> >> Eric. >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig This gets into an area that is both complicated to setup, more complicated to maintain, and harder to explain to people. I also cannot find any data on if the confusables list is whitelist or blacklist, but given it's nature of a list of characters that are confusing then I'm going to guess it's a blacklist which means it's very possible (and likely) that there are missing glyphs there that can easily be confused for each another. It also doesn't solve the problem that these names can and will be used in systems outside of a Python runtime that may or may not support unicode characters so it affords a much smaller window of compatibility. It also makes the urls a whole heck of a lot less nice. All for something that people haven't really even attempted to use (here's a total list of things that have ever been registered to PyPI with a name that uses unicode): Manual de Py2Exe en Espa??ol fl??gelform ? t? py _init__ Django D\x01jango pyramid-? The vast bulk of them being people either attempting to play with unicode or people attempting to do exactly as I outlined as a potential threat. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 15 19:12:09 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 13:12:09 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> Message-ID: <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> On May 15, 2013, at 1:07 PM, Donald Stufft wrote: > > On May 15, 2013, at 10:31 AM, Daniel Holth wrote: > >> How to avoid confusables. >> >> These scripts are recommended for use in identifiers: >> http://www.unicode.org/reports/tr31/#Table_Recommended_Scripts >> >> This report details a confusables detection algorithm: >> http://www.unicode.org/reports/tr39/#Confusable_Detection >> >> And ICU implements it: >> http://www.icu-project.org/apiref/icu4c/uspoof_8h.html (see also >> PyICU). >> >> The package index would enforce uniqueness of the "skeleton" of each >> registered package which is just an internal normalization based on >> confusability. if skeleton(identifier1) == skeleton(identifier2) then >> id1 and id2 are confusable. >> >> The tooling could get away with a simpler rule like >> re.sub("[^\w\d.]+", "_", distribution, re.UNICODE) >> >> As a bonus to including the world, this should be able to prevent >> people from exchanging zeroes for capital O. >> >> On Wed, May 15, 2013 at 7:17 AM, Eric V. Smith wrote: >>> On 05/15/2013 07:10 AM, Donald Stufft wrote: >>>>>>> Anyone want to run a scan over the PyPI package set to see >>>>>>> how many packages would cause problems for a "[a-zA-Z0-9_.-]" >>>>>>> only filter? >>>>>> >>>>>> See my previous email where I did queries against my local DB. >>>>>> It's 225 total projects that wouldn't be allowed. >>>>> >>>>> Can you send the list of those projects? >>>>> >>>>> Eric. >>>>> >>>> >>>> Here you go https://gist.github.com/dstufft/5583225 used a Python >>>> oneliner and the PyPI API so others can reproduce easily if they >>>> wish. >>> >>> Perfect. Thanks. >>> >>> It looks like space causes most of the issues. I'm not sure how >>> "Twisted Flow >= 1.0" would be expected to parse. >>> >>> Eric. >>> >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig > > > > This gets into an area that is both complicated to setup, more complicated to maintain, and harder to explain to people. > > I also cannot find any data on if the confusables list is whitelist or blacklist, but given it's nature of a list of characters that are confusing then I'm going to guess it's a blacklist which means it's very possible (and likely) that there are missing glyphs there that can easily be confused for each another. > > It also doesn't solve the problem that these names can and will be used in systems outside of a Python runtime that may or may not support unicode characters so it affords a much smaller window of compatibility. > > It also makes the urls a whole heck of a lot less nice. > > All for something that people haven't really even attempted to use (here's a total list of things that have ever been registered to PyPI with a name that uses unicode): > > Manual de Py2Exe en Espa??ol > fl??gelform > ? > t? > py > _init__ > Django > D\x01jango > pyramid-? > > The vast bulk of them being people either attempting to play with unicode or people attempting to do exactly as I outlined as a potential threat. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > It also has a problem with setuptools, distribute, and PyPI and the way they do normalization. They all already assume that projects will generally have alpha numeric names and you can take any non alpha numeric string of characters and replace it with a "-". So in order to properly support unicode you'd have to remove all the existing versions of setuptools from production use, and you'd need to update PyPI to understand how to lower case unicode. Because I registered The snowman package, you'll find it's impossible to register any other pure unicode package of any length. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 15 19:39:35 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 13:39:35 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> Message-ID: <81D08922-DE2E-4CA8-81B9-ACD05BB868FD@stufft.io> On May 15, 2013, at 1:36 PM, PJ Eby wrote: > On Wed, May 15, 2013 at 7:17 AM, Eric V. Smith wrote: >> It looks like space causes most of the issues. I'm not sure how >> "Twisted Flow >= 1.0" would be expected to parse. > > You would spell it "Twisted-Flow>=1.0", at least for setuptools-based > tools. Setuptools already restricts effective package name characters > to [a-z0-9.-], and limits the names in some other ways. For example, > you can't have multiple dashes in a row. > > Btw, I'm not sure that restricting the character set is actually going > to fix confusion problems. Donald's "Djang-" example actually works > because the command line tools interpret the request as being for a > package called "Djang-", and if I understand correctly, you would > still be able to register a package named "Djang-" on PyPI. I think > that tools will need to more explicitly address the issue of 8-bit > data in their input, e.g. by rejecting it. Yea that was coming next after/if this went through. I alluded to that by saying that tools would have a much clearer path towards what exactly a distribution name could have but yea assuming this gets accepted/goes through my next step was to open tickets in the respective tools trackers to ensure that this problem is solved there too. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Wed May 15 19:36:23 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 15 May 2013 13:36:23 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <51936ED4.2090901@trueblade.com> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> Message-ID: On Wed, May 15, 2013 at 7:17 AM, Eric V. Smith wrote: > It looks like space causes most of the issues. I'm not sure how > "Twisted Flow >= 1.0" would be expected to parse. You would spell it "Twisted-Flow>=1.0", at least for setuptools-based tools. Setuptools already restricts effective package name characters to [a-z0-9.-], and limits the names in some other ways. For example, you can't have multiple dashes in a row. Btw, I'm not sure that restricting the character set is actually going to fix confusion problems. Donald's "Djang-" example actually works because the command line tools interpret the request as being for a package called "Djang-", and if I understand correctly, you would still be able to register a package named "Djang-" on PyPI. I think that tools will need to more explicitly address the issue of 8-bit data in their input, e.g. by rejecting it. From dholth at gmail.com Wed May 15 20:10:16 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 14:10:16 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> Message-ID: On Wed, May 15, 2013 at 1:12 PM, Donald Stufft wrote: > It also has a problem with setuptools, distribute, and PyPI and the way they do normalization. They all already assume that projects will generally have alpha numeric names and you can take any non alpha numeric string of characters and replace it with a "-". So in order to properly support unicode you'd have to remove all the existing versions of setuptools from production use, and you'd need to update PyPI to understand how to lower case unicode. > > Because I registered The snowman package, you'll find it's impossible to register any other pure unicode package of any length. If PyPI has a proper i18n and Unicode implementation first, and then the tools are updated (perhaps distlib is an easier place to add Unicode than setuptools), then pypi will contain: 1. mostly ASCII projects that everyone can install 2. some Unicode projects uploaded by jerks 3. some worthwhile Unicode-named projects that might not have been uploaded before 4. some Unicode-named packages that you have to use even though you don't like the name? It's true that for a long time ASCII project names will be more convenient no matter what PyPI does, but it can be the publisher's choice rather than being cut off at the head. I don't think it's a tremendous amount of work to make Unicode work properly just for those who want it. From donald at stufft.io Wed May 15 20:33:36 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 14:33:36 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> Message-ID: On May 15, 2013, at 2:10 PM, Daniel Holth wrote: > On Wed, May 15, 2013 at 1:12 PM, Donald Stufft wrote: >> It also has a problem with setuptools, distribute, and PyPI and the way they do normalization. They all already assume that projects will generally have alpha numeric names and you can take any non alpha numeric string of characters and replace it with a "-". So in order to properly support unicode you'd have to remove all the existing versions of setuptools from production use, and you'd need to update PyPI to understand how to lower case unicode. >> >> Because I registered The snowman package, you'll find it's impossible to register any other pure unicode package of any length. > > If PyPI has a proper i18n and Unicode implementation first, and then > the tools are updated (perhaps distlib is an easier place to add > Unicode than setuptools), then pypi will contain: > > 1. mostly ASCII projects that everyone can install > > 2. some Unicode projects uploaded by jerks > > 3. some worthwhile Unicode-named projects that might not have been > uploaded before > > 4. some Unicode-named packages that you have to use even though you > don't like the name? > > It's true that for a long time ASCII project names will be more > convenient no matter what PyPI does, but it can be the publisher's > choice rather than being cut off at the head. I don't think it's a > tremendous amount of work to make Unicode work properly just for those > who want it. The problem here isn't just that the old systems won't support it. It's that they both won't support it and if someone does attempt to use a unicode package they can get an entirely different package then they expected to get. The failure case is a massive security risk. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed May 15 21:22:06 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 15:22:06 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> Message-ID: On Wed, May 15, 2013 at 2:33 PM, Donald Stufft wrote: > > On May 15, 2013, at 2:10 PM, Daniel Holth wrote: > >> On Wed, May 15, 2013 at 1:12 PM, Donald Stufft wrote: >>> It also has a problem with setuptools, distribute, and PyPI and the way they do normalization. They all already assume that projects will generally have alpha numeric names and you can take any non alpha numeric string of characters and replace it with a "-". So in order to properly support unicode you'd have to remove all the existing versions of setuptools from production use, and you'd need to update PyPI to understand how to lower case unicode. >>> >>> Because I registered The snowman package, you'll find it's impossible to register any other pure unicode package of any length. >> >> If PyPI has a proper i18n and Unicode implementation first, and then >> the tools are updated (perhaps distlib is an easier place to add >> Unicode than setuptools), then pypi will contain: >> >> 1. mostly ASCII projects that everyone can install >> >> 2. some Unicode projects uploaded by jerks >> >> 3. some worthwhile Unicode-named projects that might not have been >> uploaded before >> >> 4. some Unicode-named packages that you have to use even though you >> don't like the name? >> >> It's true that for a long time ASCII project names will be more >> convenient no matter what PyPI does, but it can be the publisher's >> choice rather than being cut off at the head. I don't think it's a >> tremendous amount of work to make Unicode work properly just for those >> who want it. > > The problem here isn't just that the old systems won't support it. It's that they both won't support it and if someone does attempt to use a unicode package they can get an entirely different package then they expected to get. The failure case is a massive security risk. Don't expose them in the simple API? If this is PyPI's big security issue then we are doing awesome. From donald at stufft.io Wed May 15 21:26:30 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 15:26:30 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> Message-ID: <6C3803CD-2312-4864-B349-91FE9C2A67CA@stufft.io> On May 15, 2013, at 3:22 PM, Daniel Holth wrote: > On Wed, May 15, 2013 at 2:33 PM, Donald Stufft wrote: >> >> On May 15, 2013, at 2:10 PM, Daniel Holth wrote: >> >>> On Wed, May 15, 2013 at 1:12 PM, Donald Stufft wrote: >>>> It also has a problem with setuptools, distribute, and PyPI and the way they do normalization. They all already assume that projects will generally have alpha numeric names and you can take any non alpha numeric string of characters and replace it with a "-". So in order to properly support unicode you'd have to remove all the existing versions of setuptools from production use, and you'd need to update PyPI to understand how to lower case unicode. >>>> >>>> Because I registered The snowman package, you'll find it's impossible to register any other pure unicode package of any length. >>> >>> If PyPI has a proper i18n and Unicode implementation first, and then >>> the tools are updated (perhaps distlib is an easier place to add >>> Unicode than setuptools), then pypi will contain: >>> >>> 1. mostly ASCII projects that everyone can install >>> >>> 2. some Unicode projects uploaded by jerks >>> >>> 3. some worthwhile Unicode-named projects that might not have been >>> uploaded before >>> >>> 4. some Unicode-named packages that you have to use even though you >>> don't like the name? >>> >>> It's true that for a long time ASCII project names will be more >>> convenient no matter what PyPI does, but it can be the publisher's >>> choice rather than being cut off at the head. I don't think it's a >>> tremendous amount of work to make Unicode work properly just for those >>> who want it. >> >> The problem here isn't just that the old systems won't support it. It's that they both won't support it and if someone does attempt to use a unicode package they can get an entirely different package then they expected to get. The failure case is a massive security risk. > > Don't expose them in the simple API? So then they are useless? It seems a lot of gotchas and gymnastics just to be academically nicer to people whose languages don't fit into ascii alpha numerics but it's only a superficial nicer since they won't actually be able to do anything useful with it. > > If this is PyPI's big security issue then we are doing awesome. This is some seriously jacked thinking and leads to nothing ever becoming secure because there's always a reason not to implement X security change because of all the other security changes needed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed May 15 21:51:46 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 15:51:46 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <6C3803CD-2312-4864-B349-91FE9C2A67CA@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> <6C3803CD-2312-4864-B349-91FE9C2A67CA@stufft.io> Message-ID: On Wed, May 15, 2013 at 3:26 PM, Donald Stufft wrote: > > On May 15, 2013, at 3:22 PM, Daniel Holth wrote: > >> On Wed, May 15, 2013 at 2:33 PM, Donald Stufft wrote: >>> >>> On May 15, 2013, at 2:10 PM, Daniel Holth wrote: >>> >>>> On Wed, May 15, 2013 at 1:12 PM, Donald Stufft wrote: >>>>> It also has a problem with setuptools, distribute, and PyPI and the way they do normalization. They all already assume that projects will generally have alpha numeric names and you can take any non alpha numeric string of characters and replace it with a "-". So in order to properly support unicode you'd have to remove all the existing versions of setuptools from production use, and you'd need to update PyPI to understand how to lower case unicode. >>>>> >>>>> Because I registered The snowman package, you'll find it's impossible to register any other pure unicode package of any length. >>>> >>>> If PyPI has a proper i18n and Unicode implementation first, and then >>>> the tools are updated (perhaps distlib is an easier place to add >>>> Unicode than setuptools), then pypi will contain: >>>> >>>> 1. mostly ASCII projects that everyone can install >>>> >>>> 2. some Unicode projects uploaded by jerks >>>> >>>> 3. some worthwhile Unicode-named projects that might not have been >>>> uploaded before >>>> >>>> 4. some Unicode-named packages that you have to use even though you >>>> don't like the name? >>>> >>>> It's true that for a long time ASCII project names will be more >>>> convenient no matter what PyPI does, but it can be the publisher's >>>> choice rather than being cut off at the head. I don't think it's a >>>> tremendous amount of work to make Unicode work properly just for those >>>> who want it. >>> >>> The problem here isn't just that the old systems won't support it. It's that they both won't support it and if someone does attempt to use a unicode package they can get an entirely different package then they expected to get. The failure case is a massive security risk. >> >> Don't expose them in the simple API? > > So then they are useless? It seems a lot of gotchas and gymnastics just to be academically nicer to people whose languages don't fit into ascii alpha numerics but it's only a superficial nicer since they won't actually be able to do anything useful with it. Not immediately. I agree with you in principle, but suggest you do something much more difficult than your original proposal by having a long-term plan for allowing Unicode in all the places while still providing for visually distinguishable names. Is anyone reading this list not an English-speaking male? Who writes malicious tutorials anyway? Wouldn't they just ask you to install a package directly from revision control? ?_? From donald at stufft.io Wed May 15 22:01:37 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 16:01:37 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> <6C3803CD-2 312-4864-B349-91FE9C2A67CA@stufft.io> Message-ID: On May 15, 2013, at 3:51 PM, Daniel Holth wrote: > > Not immediately. > > I agree with you in principle, but suggest you do something much more > difficult than your original proposal by having a long-term plan for > allowing Unicode in all the places while still providing for visually > distinguishable names. > > Is anyone reading this list not an English-speaking male? > > Who writes malicious tutorials anyway? Wouldn't they just ask you to > install a package directly from revision control? ?_? Restrictions on PyPI can always be relaxed if the tooling supports it in the future. If you want unicode everywhere you can fight the battle to get all the tooling to properly support it and get all the users and distros to upgrade their tooling to support it. I'm not willing to die on that hill because I think it's going to be a bad idea for a very long time. I care about moving us to a better state, and security and compatability trumps a facade of unicode support that quickly falls apart as soon as anyone tries to actually use it for anything serious. That's the hill I'm willing to die on. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed May 15 22:07:47 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 16:07:47 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? Message-ID: Is there an API for "all the metadata for everything" that doesn't require one web request per package version? Maybe something like an rdiff-backup of a database? From donald at stufft.io Wed May 15 22:10:47 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 16:10:47 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: <77BA6B9B-D208-425E-A98D-A61601B4E2D1@stufft.io> PyPI XMLRPC? On May 15, 2013, at 4:07 PM, Daniel Holth wrote: > Is there an API for "all the metadata for everything" that doesn't > require one web request per package version? Maybe something like an > rdiff-backup of a database? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Wed May 15 22:11:28 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 16:11:28 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: Nvm missed the one web request requirement. No I don't think so. On May 15, 2013, at 4:07 PM, Daniel Holth wrote: > Is there an API for "all the metadata for everything" that doesn't > require one web request per package version? Maybe something like an > rdiff-backup of a database? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From dholth at gmail.com Wed May 15 22:12:18 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 May 2013 16:12:18 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: Yeah, I've been using the "run bandersnatch" API, but the local storage requirement is a bit hefty. On Wed, May 15, 2013 at 4:11 PM, Donald Stufft wrote: > Nvm missed the one web request requirement. No I don't think so. > > On May 15, 2013, at 4:07 PM, Daniel Holth wrote: > >> Is there an API for "all the metadata for everything" that doesn't >> require one web request per package version? Maybe something like an >> rdiff-backup of a database? >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Wed May 15 22:13:45 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 16:13:45 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: <3562B5EE-BD20-4DB9-B2A2-1F323BE50DBF@stufft.io> What do you need it for? I could probably make a sanitized copy of crates db avail On May 15, 2013, at 4:12 PM, Daniel Holth wrote: > Yeah, I've been using the "run bandersnatch" API, but the local > storage requirement is a bit hefty. > > On Wed, May 15, 2013 at 4:11 PM, Donald Stufft wrote: >> Nvm missed the one web request requirement. No I don't think so. >> >> On May 15, 2013, at 4:07 PM, Daniel Holth wrote: >> >>> Is there an API for "all the metadata for everything" that doesn't >>> require one web request per package version? Maybe something like an >>> rdiff-backup of a database? >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Wed May 15 23:53:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 May 2013 07:53:07 +1000 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> Message-ID: One of my major goals for metadata 2.0 is better packaging interoperability with the Fedora and Debian ecosystems. Since those formally restrict the set of permitted characters, and the existing Python packaging ecosystem informally restricts it, it makes sense to limit the character set formally. By the time metadata 3.0 comes around the question may be worth revisiting, but arbitrary unicode distribution names are not going to happen for this version. Supporting them would involve a truckload of extra work in all the tools and I already expect to have to remove features from the next draft in order to make it feasible to implement in a timely fashion (Note: given that I'm not familiar with the details of the distlib, pip or setuptools code bases, I figure a "this is what I would like, now let's prune the low value hard to implement parts" is a reasonable way to get feedback from the people that are likely to end up actually implementing the changes). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrism at plope.com Thu May 16 00:08:57 2013 From: chrism at plope.com (Chris McDonough) Date: Wed, 15 May 2013 18:08:57 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <81AF75D3-519E-465D-98C7-6DA784756673@stufft.io> <510FA91B-86ED-4B19-9D40-215261F0D098@stufft.io> Message-ID: <1368655737.4035.106.camel@thinko> On Thu, 2013-05-16 at 07:53 +1000, Nick Coghlan wrote: > One of my major goals for metadata 2.0 is better packaging > interoperability with the Fedora and Debian ecosystems. Since those > formally restrict the set of permitted characters, and the existing > Python packaging ecosystem informally restricts it, it makes sense to > limit the character set formally. > > By the time metadata 3.0 comes around the question may be worth > revisiting, but arbitrary unicode distribution names are not going to > happen for this version. Supporting them would involve a truckload of > extra work in all the tools and I already expect to have to remove > features from the next draft in order to make it feasible to implement > in a timely fashion (Note: given that I'm not familiar with the > details of the distlib, pip or setuptools code bases, I figure a "this > is what I would like, now let's prune the low value hard to implement > parts" is a reasonable way to get feedback from the people that are > likely to end up actually implementing the changes). For what it's worth, I think this is a wise decision. Tool support for Unicode filenames is sorely lacking and broken. For example, git on Mac OS X can't really cope with checkouts that include filenames with high-order characters; it treats such files as having been deleted after a checkout. I checked in a single file with an accented character in a project recently, and until I removed it, it accounted for about 20% of all bug reports from people who checked things about due to spotty tool support and environmental differences. - C From berdario at gmail.com Wed May 15 16:54:19 2013 From: berdario at gmail.com (Dario Bertini) Date: Wed, 15 May 2013 16:54:19 +0200 Subject: [Distutils] Pypi Openid login problem Message-ID: Hi, I tried to login on Pypi with openid, but I keep getting back this error: on Chrome: OpenID response has been tampered with:NotAuthenticated('Replay attack detected', 9) on Firefox, strangely there's a little difference: Login failed:NotAuthenticated('Replay attack detected', 9) I tried to login from different pages, both this: https://pypi.python.org/pypi?:action=openid (which seems the plain login which I'm looking for) and this: https://pypi.python.org/pypi?:action=login&provider=Launchpad (launchpad is my Openid), after returning from launchpad, I land on a openid_return page that lets me supply some more information, (username, email and optional GPG key) with a Register button (upon clicking, I get the same error as above) Is this a known problem? thank you From nawkboy at gmail.com Wed May 15 18:08:18 2013 From: nawkboy at gmail.com (James Carpenter) Date: Wed, 15 May 2013 11:08:18 -0500 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> Message-ID: While your at it, you might consider not allowing variation in case and dash vs. underscore when specifying a dependency. A project should have only one concrete name, without fuzziness. A fuzzy match should result in a match failure. Fuzzy matches for a manual search is a different thing. On Wed, May 15, 2013 at 9:31 AM, Daniel Holth wrote: > How to avoid confusables. > > These scripts are recommended for use in identifiers: > http://www.unicode.org/reports/tr31/#Table_Recommended_Scripts > > This report details a confusables detection algorithm: > http://www.unicode.org/reports/tr39/#Confusable_Detection > > And ICU implements it: > http://www.icu-project.org/apiref/icu4c/uspoof_8h.html (see also > PyICU). > > The package index would enforce uniqueness of the "skeleton" of each > registered package which is just an internal normalization based on > confusability. if skeleton(identifier1) == skeleton(identifier2) then > id1 and id2 are confusable. > > The tooling could get away with a simpler rule like > re.sub("[^\w\d.]+", "_", distribution, re.UNICODE) > > As a bonus to including the world, this should be able to prevent > people from exchanging zeroes for capital O. > > On Wed, May 15, 2013 at 7:17 AM, Eric V. Smith wrote: > > On 05/15/2013 07:10 AM, Donald Stufft wrote: > >>>>> Anyone want to run a scan over the PyPI package set to see > >>>>> how many packages would cause problems for a "[a-zA-Z0-9_.-]" > >>>>> only filter? > >>>> > >>>> See my previous email where I did queries against my local DB. > >>>> It's 225 total projects that wouldn't be allowed. > >>> > >>> Can you send the list of those projects? > >>> > >>> Eric. > >>> > >> > >> Here you go https://gist.github.com/dstufft/5583225 used a Python > >> oneliner and the PyPI API so others can reproduce easily if they > >> wish. > > > > Perfect. Thanks. > > > > It looks like space causes most of the issues. I'm not sure how > > "Twisted Flow >= 1.0" would be expected to parse. > > > > Eric. > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j_gillick at yahoo.com Thu May 16 00:53:24 2013 From: j_gillick at yahoo.com (Jeremy Gillick) Date: Wed, 15 May 2013 15:53:24 -0700 (PDT) Subject: [Distutils] My rst README is not formatted on pypi.python.org Message-ID: <1368658404.79997.YahooMailNeo@web31806.mail.mud.yahoo.com> I wrote a valid (as far as I can tell/test)?reStructuredText README file, but on pypi it's just shown as plain text. I have run the file through the?rst2html.py tool and it works without errors and appears to be formatted correctly. Github is also able to display it correctly. Can someone tell me what I am doing wrong? My project is at: https://pypi.python.org/pypi/lcinvestor Are there any restrictions on pypi to file length of header?hierarchy? Thanks, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu May 16 02:00:53 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 20:00:53 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> Message-ID: <62CDCE56-95D9-43E2-96F2-E62EFE5224BD@stufft.io> PyPI already endures case insensitive uniqueness and considers - and _ the same for uniqueness checks On May 15, 2013, at 12:08 PM, James Carpenter wrote: > While your at it, you might consider not allowing variation in case and dash vs. underscore when specifying a dependency. A project should have only one concrete name, without fuzziness. A fuzzy match should result in a match failure. Fuzzy matches for a manual search is a different thing. > > > On Wed, May 15, 2013 at 9:31 AM, Daniel Holth wrote: >> How to avoid confusables. >> >> These scripts are recommended for use in identifiers: >> http://www.unicode.org/reports/tr31/#Table_Recommended_Scripts >> >> This report details a confusables detection algorithm: >> http://www.unicode.org/reports/tr39/#Confusable_Detection >> >> And ICU implements it: >> http://www.icu-project.org/apiref/icu4c/uspoof_8h.html (see also >> PyICU). >> >> The package index would enforce uniqueness of the "skeleton" of each >> registered package which is just an internal normalization based on >> confusability. if skeleton(identifier1) == skeleton(identifier2) then >> id1 and id2 are confusable. >> >> The tooling could get away with a simpler rule like >> re.sub("[^\w\d.]+", "_", distribution, re.UNICODE) >> >> As a bonus to including the world, this should be able to prevent >> people from exchanging zeroes for capital O. >> >> On Wed, May 15, 2013 at 7:17 AM, Eric V. Smith wrote: >> > On 05/15/2013 07:10 AM, Donald Stufft wrote: >> >>>>> Anyone want to run a scan over the PyPI package set to see >> >>>>> how many packages would cause problems for a "[a-zA-Z0-9_.-]" >> >>>>> only filter? >> >>>> >> >>>> See my previous email where I did queries against my local DB. >> >>>> It's 225 total projects that wouldn't be allowed. >> >>> >> >>> Can you send the list of those projects? >> >>> >> >>> Eric. >> >>> >> >> >> >> Here you go https://gist.github.com/dstufft/5583225 used a Python >> >> oneliner and the PyPI API so others can reproduce easily if they >> >> wish. >> > >> > Perfect. Thanks. >> > >> > It looks like space causes most of the issues. I'm not sure how >> > "Twisted Flow >= 1.0" would be expected to parse. >> > >> > Eric. >> > >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > http://mail.python.org/mailman/listinfo/distutils-sig >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Thu May 16 02:21:36 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 15 May 2013 17:21:36 -0700 Subject: [Distutils] My rst README is not formatted on pypi.python.org In-Reply-To: <1368658404.79997.YahooMailNeo@web31806.mail.mud.yahoo.com> References: <1368658404.79997.YahooMailNeo@web31806.mail.mud.yahoo.com> Message-ID: On Wed, May 15, 2013 at 3:53 PM, Jeremy Gillick wrote: > I wrote a valid (as far as I can tell/test) reStructuredText README file, > but on pypi it's just shown as plain text. I have run the file through the > rst2html.py tool and it works without errors and appears to be formatted > correctly. Github is also able to display it correctly. Can someone tell me > what I am doing wrong? Did you try the --no-raw option as suggested here? http://docs.python.org/2/distutils/packageindex.html#pypi-package-display In the past I filed an issue about what you are experiencing because I suspect it is a common problem. The issue is here: https://sourceforge.net/p/pypi/bugs/360/ --Chris > My project is at: > > https://pypi.python.org/pypi/lcinvestor > > Are there any restrictions on pypi to file length of header hierarchy? > > Thanks, > Jeremy > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From ncoghlan at gmail.com Thu May 16 02:35:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 May 2013 10:35:49 +1000 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: <62CDCE56-95D9-43E2-96F2-E62EFE5224BD@stufft.io> References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <62CDCE56-95D9-43E2-96F2-E62EFE5224BD@stufft.io> Message-ID: On Thu, May 16, 2013 at 10:00 AM, Donald Stufft wrote: > PyPI already endures case insensitive uniqueness and considers - and _ the > same for uniqueness checks Still something we should formalise in the 2.0 spec, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Thu May 16 02:59:08 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 20:59:08 -0400 Subject: [Distutils] Proposal: Restrict the characters in a project name In-Reply-To: References: <1A35F9C4-5EC7-4033-BFEE-C2A2C760E63D@stufft.io> <5C2CD853-6904-4095-9472-4D823594EC9A@stufft.io> <0DD4D8FB-0945-4A0F-913B-494B4ED25F90@stufft.io> <73794147-A7B6-417B-B819-4899DDBFA850@stufft.io> <148FE34E-58C6-4C5F-93BF-EAD002A9E85E@coderanger.net> <80B87007-1CCA-42AC-81E5-801C31F30EF3@stufft.io> <48B812CE-F3EE-489A-A7F3-534A8B45D598@trueblade.com> <40B5A52C-5EF4-439E-91E0-7052DEC36F24@stufft.io> <51936ED4.2090901@trueblade.com> <62CDCE56-95D9-43E2-96F2-E62EFE5224BD@stufft.io> Message-ID: <28F1286D-734C-46F1-B42D-AD75FA255CFD@stufft.io> On May 15, 2013, at 8:35 PM, Nick Coghlan wrote: > On Thu, May 16, 2013 at 10:00 AM, Donald Stufft wrote: >> PyPI already endures case insensitive uniqueness and considers - and _ the >> same for uniqueness checks > > Still something we should formalise in the 2.0 spec, though. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia The rule would be case insensitive, alpha numerics, and period, hyphen, and underscore where hypen and underscore are considered equivalent. Just verified this based on the current code + the proposal. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 16 03:01:05 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 May 2013 21:01:05 -0400 Subject: [Distutils] My rst README is not formatted on pypi.python.org In-Reply-To: <1368658404.79997.YahooMailNeo@web31806.mail.mud.yahoo.com> References: <1368658404.79997.YahooMailNeo@web31806.mail.mud.yahoo.com> Message-ID: <9DF74A58-34B0-407F-B4B9-A3CA8DF5B7F4@stufft.io> On May 15, 2013, at 6:53 PM, Jeremy Gillick wrote: > I wrote a valid (as far as I can tell/test) reStructuredText README file, but on pypi it's just shown as plain text. I have run the file through the rst2html.py tool and it works without errors and appears to be formatted correctly. Github is also able to display it correctly. Can someone tell me what I am doing wrong? My project is at: > > https://pypi.python.org/pypi/lcinvestor > > Are there any restrictions on pypi to file length of header hierarchy? > > Thanks, > Jeremy > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig It's probably due to your use of urls like #minmax-percent-interest-rate, PyPI whitelists the allowed url schemes and I don't think "no scheme" is an allowed one. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From j_gillick at yahoo.com Thu May 16 02:48:49 2013 From: j_gillick at yahoo.com (Jeremy Gillick) Date: Wed, 15 May 2013 17:48:49 -0700 (PDT) Subject: [Distutils] My rst README is not formatted on pypi.python.org In-Reply-To: References: <1368658404.79997.YahooMailNeo@web31806.mail.mud.yahoo.com> Message-ID: <1368665329.71088.YahooMailNeo@web31809.mail.mud.yahoo.com> >?Did you try the --no-raw option as suggested here? I just tried that and it still outputs the HTML version without any errors and the only warning is: "UserWarning: Unknown distribution option: 'install_requires'" -- but that seems unrelated. Your bug seems very related to what I'm experiencing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Thu May 16 15:31:08 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 16 May 2013 09:31:08 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Tue, May 14, 2013 at 10:36 PM, Daniel Holth wrote: > Why don't we simply provide an option to ignore == if it is not a > "root" dependency? I don't understand what you mean. Jim > > On Mon, May 13, 2013 at 2:02 PM, Jim Fulton wrote: >> On Mon, May 13, 2013 at 10:16 AM, Ian Cordasco >> wrote: >>> On Mon, May 13, 2013 at 10:12 AM, Reinout van Rees wrote: >>>> On 13-05-13 15:57, Ian Cordasco wrote: >>>>> >>>>> If I release a library dependent upon a particular API in one version >>>>> of a dependency and before I release my next version I notice plans to >>>>> break the existing API, why shouldn't I pin the version to protect >>>>> users (or at least provide a maximum version) from getting horrible >>>>> exceptions? >>>> >>>> >>>> Best example: if you pin "somelibrary=1.2" in your library, none of your >>>> users can use the critical 1.2.1 bugfix release. >>> >>> Thanks to you and Daniel. (I accidentally replied to Daniel off-list.) >>> I had just been advised by someone formerly a part of the distutils >>> (or distribute, I forget which) team that you either pin or don't. I >>> try not to, but there have been occasions where I found it necessary. >>> I'll certainly move forward a better developer for your explanations. >> >> Unfortunately, when people discuss solutions, they often >> forget to state the problem they're thinking of. >> >> As Daniel pointed out, when deploying an *application*, you should >> generally pin all of your dependencies, so you can reproduce the >> deployment. This can be done via the apps setup.cfg, pip requirements >> or buildout versions. >> >> When releasing libraries, you should restrict versions as little as >> possible. The more you restrict dependency versions, the harder your >> library is to use. A common recommendation is to set a minimum version >> to the lowest version of the dependency known to work and to set the >> maximum version to less then the next major release: >> >> >=2, <3dev >> >> In practice, most people just set a lower bound, if any. >> >> Jim >> >> -- >> Jim Fulton >> http://www.linkedin.com/in/jimfulton >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton http://www.linkedin.com/in/jimfulton From dholth at gmail.com Thu May 16 15:51:42 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 16 May 2013 09:51:42 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: If you were to say: install gerbil==3 wheel==0.16 and gerbil version 3's requirements were: water_bottle == 4 shavings < 7 wheel >= 0.16 # of course and shavings's requirements were: cedar == 0.9 The root of the dependency graph is "gerbil==3 wheel==0.16.0". These are the only == constraints that will be honored. The proposed option would keep "gerbil==3 wheel==0.16.0", convert water_bottle==4 and cedar==0.9 to just water_bottle and cedar, and respect the >= and < constraints. On Thu, May 16, 2013 at 9:31 AM, Jim Fulton wrote: > On Tue, May 14, 2013 at 10:36 PM, Daniel Holth wrote: >> Why don't we simply provide an option to ignore == if it is not a >> "root" dependency? > > I don't understand what you mean. > > Jim > >> >> On Mon, May 13, 2013 at 2:02 PM, Jim Fulton wrote: >>> On Mon, May 13, 2013 at 10:16 AM, Ian Cordasco >>> wrote: >>>> On Mon, May 13, 2013 at 10:12 AM, Reinout van Rees wrote: >>>>> On 13-05-13 15:57, Ian Cordasco wrote: >>>>>> >>>>>> If I release a library dependent upon a particular API in one version >>>>>> of a dependency and before I release my next version I notice plans to >>>>>> break the existing API, why shouldn't I pin the version to protect >>>>>> users (or at least provide a maximum version) from getting horrible >>>>>> exceptions? >>>>> >>>>> >>>>> Best example: if you pin "somelibrary=1.2" in your library, none of your >>>>> users can use the critical 1.2.1 bugfix release. >>>> >>>> Thanks to you and Daniel. (I accidentally replied to Daniel off-list.) >>>> I had just been advised by someone formerly a part of the distutils >>>> (or distribute, I forget which) team that you either pin or don't. I >>>> try not to, but there have been occasions where I found it necessary. >>>> I'll certainly move forward a better developer for your explanations. >>> >>> Unfortunately, when people discuss solutions, they often >>> forget to state the problem they're thinking of. >>> >>> As Daniel pointed out, when deploying an *application*, you should >>> generally pin all of your dependencies, so you can reproduce the >>> deployment. This can be done via the apps setup.cfg, pip requirements >>> or buildout versions. >>> >>> When releasing libraries, you should restrict versions as little as >>> possible. The more you restrict dependency versions, the harder your >>> library is to use. A common recommendation is to set a minimum version >>> to the lowest version of the dependency known to work and to set the >>> maximum version to less then the next major release: >>> >>> >=2, <3dev >>> >>> In practice, most people just set a lower bound, if any. >>> >>> Jim >>> >>> -- >>> Jim Fulton >>> http://www.linkedin.com/in/jimfulton >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig > > > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton From donald at stufft.io Thu May 16 15:57:49 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 16 May 2013 09:57:49 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: <47EB92AD-D440-43E6-B148-F73F23923815@stufft.io> Pip respects == but requirements.txt can override. On May 16, 2013, at 9:51 AM, Daniel Holth wrote: > If you were to say: > > install gerbil==3 wheel==0.16 > > and gerbil version 3's requirements were: > > water_bottle == 4 > shavings < 7 > wheel >= 0.16 # of course > > and shavings's requirements were: > > cedar == 0.9 > > The root of the dependency graph is "gerbil==3 wheel==0.16.0". These > are the only == constraints that will be honored. > > The proposed option would keep "gerbil==3 wheel==0.16.0", convert > water_bottle==4 and cedar==0.9 to just water_bottle and cedar, and > respect the >= and < constraints. > > > On Thu, May 16, 2013 at 9:31 AM, Jim Fulton wrote: >> On Tue, May 14, 2013 at 10:36 PM, Daniel Holth wrote: >>> Why don't we simply provide an option to ignore == if it is not a >>> "root" dependency? >> >> I don't understand what you mean. >> >> Jim >> >>> >>> On Mon, May 13, 2013 at 2:02 PM, Jim Fulton wrote: >>>> On Mon, May 13, 2013 at 10:16 AM, Ian Cordasco >>>> wrote: >>>>> On Mon, May 13, 2013 at 10:12 AM, Reinout van Rees wrote: >>>>>> On 13-05-13 15:57, Ian Cordasco wrote: >>>>>>> >>>>>>> If I release a library dependent upon a particular API in one version >>>>>>> of a dependency and before I release my next version I notice plans to >>>>>>> break the existing API, why shouldn't I pin the version to protect >>>>>>> users (or at least provide a maximum version) from getting horrible >>>>>>> exceptions? >>>>>> >>>>>> >>>>>> Best example: if you pin "somelibrary=1.2" in your library, none of your >>>>>> users can use the critical 1.2.1 bugfix release. >>>>> >>>>> Thanks to you and Daniel. (I accidentally replied to Daniel off-list.) >>>>> I had just been advised by someone formerly a part of the distutils >>>>> (or distribute, I forget which) team that you either pin or don't. I >>>>> try not to, but there have been occasions where I found it necessary. >>>>> I'll certainly move forward a better developer for your explanations. >>>> >>>> Unfortunately, when people discuss solutions, they often >>>> forget to state the problem they're thinking of. >>>> >>>> As Daniel pointed out, when deploying an *application*, you should >>>> generally pin all of your dependencies, so you can reproduce the >>>> deployment. This can be done via the apps setup.cfg, pip requirements >>>> or buildout versions. >>>> >>>> When releasing libraries, you should restrict versions as little as >>>> possible. The more you restrict dependency versions, the harder your >>>> library is to use. A common recommendation is to set a minimum version >>>> to the lowest version of the dependency known to work and to set the >>>> maximum version to less then the next major release: >>>> >>>>> =2, <3dev >>>> >>>> In practice, most people just set a lower bound, if any. >>>> >>>> Jim >>>> >>>> -- >>>> Jim Fulton >>>> http://www.linkedin.com/in/jimfulton >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> http://mail.python.org/mailman/listinfo/distutils-sig >> >> >> >> -- >> Jim Fulton >> http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From dholth at gmail.com Thu May 16 16:13:45 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 16 May 2013 10:13:45 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Thu, May 16, 2013 at 10:08 AM, Jim Fulton wrote: > On Thu, May 16, 2013 at 9:51 AM, Daniel Holth wrote: >> If you were to say: >> >> install gerbil==3 wheel==0.16 >> >> and gerbil version 3's requirements were: >> >> water_bottle == 4 >> shavings < 7 >> wheel >= 0.16 # of course >> >> and shavings's requirements were: >> >> cedar == 0.9 >> >> The root of the dependency graph is "gerbil==3 wheel==0.16.0". These >> are the only == constraints that will be honored. >> >> The proposed option would keep "gerbil==3 wheel==0.16.0", convert >> water_bottle==4 and cedar==0.9 to just water_bottle and cedar, and >> respect the >= and < constraints. > > Ignoring requirements, especially in the absence of conflict, as in > the case above, seems like a bad idea to me. I could see high-level > == requirements overriding lower-level requirements, possibly with a > warning. > > (I'm not sure what the scope of this discussion is; whether it's just > pip, or whether > the original question was meant to be general.) It is a general idea which would of course have to be implemented in pip or whatever. Generally, what do do when your dependencies declare incorrect dependencies of any kind; specifically the idea of making it easy to ignore == because it is almost always wrong. From jim at zope.com Thu May 16 16:08:42 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 16 May 2013 10:08:42 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Thu, May 16, 2013 at 9:51 AM, Daniel Holth wrote: > If you were to say: > > install gerbil==3 wheel==0.16 > > and gerbil version 3's requirements were: > > water_bottle == 4 > shavings < 7 > wheel >= 0.16 # of course > > and shavings's requirements were: > > cedar == 0.9 > > The root of the dependency graph is "gerbil==3 wheel==0.16.0". These > are the only == constraints that will be honored. > > The proposed option would keep "gerbil==3 wheel==0.16.0", convert > water_bottle==4 and cedar==0.9 to just water_bottle and cedar, and > respect the >= and < constraints. Ignoring requirements, especially in the absence of conflict, as in the case above, seems like a bad idea to me. I could see high-level == requirements overriding lower-level requirements, possibly with a warning. (I'm not sure what the scope of this discussion is; whether it's just pip, or whether the original question was meant to be general.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Thu May 16 16:16:17 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 16 May 2013 10:16:17 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: On Thu, May 16, 2013 at 10:13 AM, Daniel Holth wrote: > On Thu, May 16, 2013 at 10:08 AM, Jim Fulton wrote: >> On Thu, May 16, 2013 at 9:51 AM, Daniel Holth wrote: >>> If you were to say: >>> >>> install gerbil==3 wheel==0.16 >>> >>> and gerbil version 3's requirements were: >>> >>> water_bottle == 4 >>> shavings < 7 >>> wheel >= 0.16 # of course >>> >>> and shavings's requirements were: >>> >>> cedar == 0.9 >>> >>> The root of the dependency graph is "gerbil==3 wheel==0.16.0". These >>> are the only == constraints that will be honored. >>> >>> The proposed option would keep "gerbil==3 wheel==0.16.0", convert >>> water_bottle==4 and cedar==0.9 to just water_bottle and cedar, and >>> respect the >= and < constraints. >> >> Ignoring requirements, especially in the absence of conflict, as in >> the case above, seems like a bad idea to me. I could see high-level >> == requirements overriding lower-level requirements, possibly with a >> warning. >> >> (I'm not sure what the scope of this discussion is; whether it's just >> pip, or whether >> the original question was meant to be general.) > > It is a general idea which would of course have to be implemented in > pip or whatever. Generally, what do do when your dependencies > declare incorrect dependencies of any kind; specifically the idea of > making it easy to ignore == because it is almost always wrong. I don't think it should be up to the tool to decide that a dependency is wrong. IMO, the tool should satisfy the declared dependencies as well as possible, report conflicts, and give the user a way to decide a conflict. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From dholth at gmail.com Thu May 16 16:19:32 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 16 May 2013 10:19:32 -0400 Subject: [Distutils] Overriding dependency versions In-Reply-To: References: Message-ID: > I don't think it should be up to the tool to decide that a dependency is wrong. > > IMO, the tool should satisfy the declared dependencies as well as possible, > report conflicts, and give the user a way to decide a conflict. In this case we don't expect a conflict, but are unfortunate enough to be using a dependency that pins some other project. The tool would ideally warn us that there was a newer version of some pinned dependency on an interior node of our dependency graph and let us do something about it. From marius at pov.lt Thu May 16 20:23:58 2013 From: marius at pov.lt (Marius Gedminas) Date: Thu, 16 May 2013 21:23:58 +0300 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <77BA6B9B-D208-425E-A98D-A61601B4E2D1@stufft.io> References: <77BA6B9B-D208-425E-A98D-A61601B4E2D1@stufft.io> Message-ID: <20130516182358.GA6170@fridge.pov.lt> On Wed, May 15, 2013 at 04:10:47PM -0400, Donald Stufft wrote: > PyPI XMLRPC? Doesn't that require *two* HTTP requests per package? One to get a list of package versions, and one to get the metadata for a specified version number? I studied http://wiki.python.org/moin/PyPIXmlRpc as best as I could before deciding I couldn't do any better than one HTTP request per package (to get JSON data for the latest release) for https://github.com/mgedmin/ztk-py3-status/blob/master/get_pypi_status.py Marius Gedminas -- Never trust a smiling Gates. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From dw at botanicus.net Thu May 16 21:46:04 2013 From: dw at botanicus.net (David Wilson) Date: Thu, 16 May 2013 20:46:04 +0100 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: Would something like http://pypi.h1.botanicus.net/static/dump.txt.gz be useful to you? (warning: 57mb expanding to 540mb). Each line is a JSON-encoded dict containing a single package release. for line in gzip.open('dump.txt.gz'): dct = json.loads(line) .... etc The code for it is very simple, would be willing to clean it up and turn it into a cron job if people found it useful. Note the dump above is outdated, I only made it as a test. On 15 May 2013 21:12, Daniel Holth wrote: > Yeah, I've been using the "run bandersnatch" API, but the local > storage requirement is a bit hefty. > > On Wed, May 15, 2013 at 4:11 PM, Donald Stufft wrote: > > Nvm missed the one web request requirement. No I don't think so. > > > > On May 15, 2013, at 4:07 PM, Daniel Holth wrote: > > > >> Is there an API for "all the metadata for everything" that doesn't > >> require one web request per package version? Maybe something like an > >> rdiff-backup of a database? > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> http://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nawkboy+distutils at gmail.com Thu May 16 21:52:07 2013 From: nawkboy+distutils at gmail.com (James Carpenter) Date: Thu, 16 May 2013 14:52:07 -0500 Subject: [Distutils] Continuous Deployment Style Build System for Python (Requesting Feedback) Message-ID: Please take a look and let us know what you think. Please include feedback on any confusion or errors in the docs too, so we can fix them. =========================================== Defend Against Fruit is focused on providing a pragmatic, continuous deployment style build system for Python. Current Python build systems do not properly account for the needs of effective continuous deployment. This package extends the Python tooling to add the missing pieces. With an eye to agile development principles and fast-feedback, we want a build system which satisfies the following goals: * Every SCM change-set committed should result in a potentially shippable release candidate. * When a defect is introduced, we want to immediately detect and isolate the offending SCM change-set. This is true even if the defect was introduced into a library we depend upon. * Library management should be so easy as to never impede code changes, even in multi-component architecture. More details available at: http://teamfruit.github.io/defend_against_fruit/ License: Apache Public License v2 Authors: James Carpenter jcarpenter621 at yahoo.com LinkedIn: http://www.linkedin.com/in/jamescarpenter1 Matthew Tardiff mattrix at gmail.com LinkedIn: http://www.linkedin.com/in/matthewtardiff -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 16 23:23:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 May 2013 07:23:03 +1000 Subject: [Distutils] Continuous Deployment Style Build System for Python (Requesting Feedback) In-Reply-To: References: Message-ID: Very interesting! The next draft of the metadata 2.0 spec is probably a couple of weeks away from broader public consumption, at which time I'll be interested in hearing whether or not that better meets DAF's needs (especially for transitive dependency tracking). (I haven't been putting the interim PEP 426 updates on python.org as they're currently a somewhat incoherent mixture of the first draft of the new JSON based interchange format and the rationale for the last version that still used the old key:value format) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From j_gillick at yahoo.com Thu May 16 20:16:07 2013 From: j_gillick at yahoo.com (Jeremy Gillick) Date: Thu, 16 May 2013 11:16:07 -0700 (PDT) Subject: [Distutils] My rst README is not formatted on pypi.python.org In-Reply-To: <9DF74A58-34B0-407F-B4B9-A3CA8DF5B7F4@stufft.io> References: <1368658404.79997.YahooMailNeo@web31806.mail.mud.yahoo.com> <9DF74A58-34B0-407F-B4B9-A3CA8DF5B7F4@stufft.io> Message-ID: <1368728167.11414.YahooMailNeo@web31801.mail.mud.yahoo.com> That fixed it! Which is kinda silly that in-page links would be filtered out. In any case, thanks for helping me to get this working! ________________________________ From: Donald Stufft? To: Jeremy Gillick? Sent: Wednesday, May 15, 2013 6:01 PM Subject: Re: [Distutils] My rst README is not formatted on pypi.python.org On May 15, 2013, at 6:53 PM, Jeremy Gillick wrote: I wrote a valid (as far as I can tell/test)?reStructuredText README file, but on pypi it's just shown as plain text. I have run the file through the?rst2html.py tool and it works without errors and appears to be formatted correctly. Github is also able to display it correctly. Can someone tell me what I am doing wrong? My project is at: > > >https://pypi.python.org/pypi/lcinvestor > > > >Are there any restrictions on pypi to file length of header?hierarchy? > > >Thanks, >Jeremy_______________________________________________ >Distutils-SIG maillist ?- ?Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig > It's probably due to your use of urls like?#minmax-percent-interest-rate, PyPI whitelists the allowed url schemes and I don't think "no scheme" is an allowed one. ----------------- Donald Stufft PGP:?0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From nawkboy at gmail.com Thu May 16 21:08:16 2013 From: nawkboy at gmail.com (James Carpenter) Date: Thu, 16 May 2013 14:08:16 -0500 Subject: [Distutils] Continuous Deployment Style Build System for Python Message-ID: Defend Against Fruit is focused on providing a pragmatic, continuous deployment style build system for Python. Current Python build systems do not properly account for the needs of effective continuous deployment. This package extends the Python tooling to add the missing pieces. With an eye to agile development principles and fast-feedback, we want a build system which satisfies the following goals: * Every SCM change-set committed should result in a potentially shippable release candidate. * When a defect is introduced, we want to immediately detect and isolate the offending SCM change-set. This is true even if the defect was introduced into a library we depend upon. * Library management should be so easy as to never impede code changes, even in multi-component architecture. More details available at: http://teamfruit.github.io/defend_against_fruit/ License: Apache Public License v2 Authors: James Carpenter jcarpenter621 at yahoo.com LinkedIn: http://www.linkedin.com/in/jamescarpenter1 Matthew Tardiff mattrix at gmail.com LinkedIn: http://www.linkedin.com/in/matthewtardiff -------------- next part -------------- An HTML attachment was scrubbed... URL: From nawkboy at gmail.com Fri May 17 00:55:20 2013 From: nawkboy at gmail.com (James Carpenter) Date: Thu, 16 May 2013 17:55:20 -0500 Subject: [Distutils] Continuous Deployment Style Build System for Python (Requesting Feedback) In-Reply-To: References: Message-ID: I haven't read PEP 426, but one of the things I would keep in mind is to consider moving to or additionally supporting a Maven style repository layout. It isn't that a Maven layout is necessary better than layout X. It is just that the Maven artifact repository managers are far more mature than anything we saw in PyPI land and probably will be for years to come. https://github.com/teamfruit/defend_against_fruit/wiki/Survey-of-Existing-PyPI-Implementations This is probably largely orthogonal to how your dependency meta-data is maintained. The only real constraint would be to ensure the meta-data file is published alongside the artifact (similar to ivy.xml or pom.xml files). Otherwise, the only efficient way to navigate the meta-data is to bake the functionality into the repository and expose a web service. That isn't necessary wrong in theory, but in practice it would mean you couldn't just use a mature Maven derived repository manager like Nexus, Artifactory, or Archiva. In both Maven and Ivy the repository can be rather stupid since the dependency resolution mechanisms are baked into the build tool instead of the repository. On Thu, May 16, 2013 at 4:23 PM, Nick Coghlan wrote: > Very interesting! > > The next draft of the metadata 2.0 spec is probably a couple of weeks away > from broader public consumption, at which time I'll be interested in > hearing whether or not that better meets DAF's needs (especially for > transitive dependency tracking). > > (I haven't been putting the interim PEP 426 updates on python.org as > they're currently a somewhat incoherent mixture of the first draft of the > new JSON based interchange format and the rationale for the last version > that still used the old key:value format) > > Cheers, > Nick. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri May 17 02:17:20 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 16 May 2013 20:17:20 -0400 Subject: [Distutils] Continuous Deployment Style Build System for Python (Requesting Feedback) In-Reply-To: References: Message-ID: You will be astonished to learn exactly how dumb the index is. Most of the metadata used for package discovery is in the file names. Then whole packages are downloaded and executed to produce their metadata. Improving this is a major goal. On May 16, 2013 7:35 PM, "James Carpenter" wrote: > I haven't read PEP 426, but one of the things I would keep in mind is to > consider moving to or additionally supporting a Maven style repository > layout. It isn't that a Maven layout is necessary better than layout X. It > is just that the Maven artifact repository managers are far more mature > than anything we saw in PyPI land and probably will be for years to come. > https://github.com/teamfruit/defend_against_fruit/wiki/Survey-of-Existing-PyPI-Implementations > > This is probably largely orthogonal to how your dependency meta-data is > maintained. The only real constraint would be to ensure the meta-data file > is published alongside the artifact (similar to ivy.xml or pom.xml files). > Otherwise, the only efficient way to navigate the meta-data is to bake the > functionality into the repository and expose a web service. That isn't > necessary wrong in theory, but in practice it would mean you couldn't just > use a mature Maven derived repository manager like Nexus, Artifactory, or > Archiva. In both Maven and Ivy the repository can be rather stupid since > the dependency resolution mechanisms are baked into the build tool instead > of the repository. > > > On Thu, May 16, 2013 at 4:23 PM, Nick Coghlan wrote: > >> Very interesting! >> >> The next draft of the metadata 2.0 spec is probably a couple of weeks >> away from broader public consumption, at which time I'll be interested in >> hearing whether or not that better meets DAF's needs (especially for >> transitive dependency tracking). >> >> (I haven't been putting the interim PEP 426 updates on python.org as >> they're currently a somewhat incoherent mixture of the first draft of the >> new JSON based interchange format and the rationale for the last version >> that still used the old key:value format) >> >> Cheers, >> Nick. >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri May 17 03:50:17 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 16 May 2013 21:50:17 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: On Thu, May 16, 2013 at 3:46 PM, David Wilson wrote: > Would something like http://pypi.h1.botanicus.net/static/dump.txt.gz be > useful to you? (warning: 57mb expanding to 540mb). Each line is a > JSON-encoded dict containing a single package release. > > for line in gzip.open('dump.txt.gz'): > dct = json.loads(line) > .... > > etc > > The code for it is very simple, would be willing to clean it up and turn it > into a cron job if people found it useful. > > Note the dump above is outdated, I only made it as a test. Seems like a useful format. https://bitbucket.org/dholth/pypi_stats is a prototype that parses requires.txt and other metadata out of all the sdists in a folder, putting them into a sqlite3 database. It may be interesting for experimentation. For example, I can easily tell you how many different version numbers there are and which are the most popular, or I can tell you which metadata keys and version numbers have been used. The database winds up being 1.6 GB or about 200MB if you delete the unparsed files. From dw at botanicus.net Fri May 17 03:55:39 2013 From: dw at botanicus.net (David Wilson) Date: Fri, 17 May 2013 02:55:39 +0100 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: Interesting! I produced that dump as part of a demo of using Xapian for cheese shop search (still a work in progress, when I get a free moment). Adding e.g. a "depends:" operator is something I'd like, and your database sounds very useful for achieving that goal. Thanks for the link. I may be e-mailing you shortly ;) On 17 May 2013 02:50, Daniel Holth wrote: > On Thu, May 16, 2013 at 3:46 PM, David Wilson wrote: > > Would something like http://pypi.h1.botanicus.net/static/dump.txt.gz be > > useful to you? (warning: 57mb expanding to 540mb). Each line is a > > JSON-encoded dict containing a single package release. > > > > for line in gzip.open('dump.txt.gz'): > > dct = json.loads(line) > > .... > > > > etc > > > > The code for it is very simple, would be willing to clean it up and turn > it > > into a cron job if people found it useful. > > > > Note the dump above is outdated, I only made it as a test. > > Seems like a useful format. > > https://bitbucket.org/dholth/pypi_stats is a prototype that parses > requires.txt and other metadata out of all the sdists in a folder, > putting them into a sqlite3 database. It may be interesting for > experimentation. For example, I can easily tell you how many different > version numbers there are and which are the most popular, or I can > tell you which metadata keys and version numbers have been used. The > database winds up being 1.6 GB or about 200MB if you delete the > unparsed files. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reachtotj at gmail.com Fri May 17 17:20:36 2013 From: reachtotj at gmail.com (Taranjeet Singh) Date: Fri, 17 May 2013 20:50:36 +0530 Subject: [Distutils] problem Message-ID: neither distribute nor pip is installing on my laptop (windows). installing distribute gives an error of Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.39.tar.gz Extracting in c:\users\jatin\appdata\local\temp\tmpec1sv7 Now working in c:\users\jatin\appdata\local\temp\tmpec1sv7\distribute-0.6.39 Installing Distribute Something went wrong during the installation. See the error message above. Traceback (most recent call last): File "C:\Python27\distribute_setup.py", line 546, in sys.exit(main()) SystemExit: 2 please help me out -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Sat May 18 00:29:54 2013 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 17 May 2013 18:29:54 -0400 Subject: [Distutils] 504 timeouts on pypi.python.org Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Both in the web interface and in registration / upload of releases. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGWr2IACgkQ+gerLs4ltQ5bAwCffg+Zn0MwHPaxrALbCSbHXeRP 1LEAoKMyu/z9nb1+nBSktEE0spAWYQ3W =QIvr -----END PGP SIGNATURE----- From vinay_sajip at yahoo.co.uk Sat May 18 02:16:31 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 18 May 2013 00:16:31 +0000 (UTC) Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? References: Message-ID: Daniel Holth gmail.com> writes: > Is there an API for "all the metadata for everything" that doesn't > require one web request per package version? Maybe something like an > rdiff-backup of a database? Well, the red-dove.com metadata works at three levels: http://www.red-dove.com/pypi/projects/projects.json gives a list of all PYPI project names, then e.g. http://www.red-dove.com/pypi/projects/W/wheel/project.json gives all the version numbers for wheel with download URLs, MD5 hashes (and importantly, dependencies) and e.g. http://www.red-dove.com/pypi/projects/W/wheel/package-1.0.0a2.json will give all the particulars of that particular wheel version, including dependencies, sources, build and test options, package data etc. So, the middle level gives a useful overview of a project sufficient for dependency resolution (it's what distlib and distil use). The data is kept reasonably fresh by periodically looking at what's changed recently on PyPI and updating the metadata for new releases etc. Regards, Vinay Sajip From chris at kateandchris.net Sat May 18 04:24:16 2013 From: chris at kateandchris.net (Chris Lambacher) Date: Fri, 17 May 2013 22:24:16 -0400 Subject: [Distutils] problem In-Reply-To: References: Message-ID: Are you using an Administrator Terminal? How are you attempting to install them? -Chris On Fri, May 17, 2013 at 11:20 AM, Taranjeet Singh wrote: > neither distribute nor pip is installing on my laptop (windows). > installing distribute gives an error of > Downloading > http://pypi.python.org/packages/source/d/distribute/distribute-0.6.39.tar.gz > Extracting in c:\users\jatin\appdata\local\temp\tmpec1sv7 > Now working in > c:\users\jatin\appdata\local\temp\tmpec1sv7\distribute-0.6.39 > Installing Distribute > Something went wrong during the installation. > See the error message above. > > Traceback (most recent call last): > File "C:\Python27\distribute_setup.py", line 546, in > sys.exit(main()) > SystemExit: 2 > please help me out > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -- Christopher Lambacher chris at kateandchris.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat May 18 21:16:46 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 18 May 2013 15:16:46 -0400 Subject: [Distutils] 504 timeouts on pypi.python.org In-Reply-To: References: Message-ID: On May 17, 2013, at 6:29 PM, Tres Seaver wrote: > Signed PGP part > Both in the web interface and in registration / upload of releases. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Is this still happening to you? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tseaver at palladion.com Sat May 18 23:10:54 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 18 May 2013 17:10:54 -0400 Subject: [Distutils] 504 timeouts on pypi.python.org In-Reply-To: References: Message-ID: <5197EE5E.4080900@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/18/2013 03:16 PM, Donald Stufft wrote: > > On May 17, 2013, at 6:29 PM, Tres Seaver > wrote: > >> Both in the web interface and in registration / upload of releases. >> > Is this still happening to you? This afternoon I was able to register / upload the release for which I noticed the problem yesterday. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGX7l4ACgkQ+gerLs4ltQ6hLgCgw0eK/PNjmAWz12+LZqnLQfJt TgYAoIzreHKL8WxXAzNMEyyNBuknC7h/ =djJM -----END PGP SIGNATURE----- From richard at python.org Sun May 19 01:00:14 2013 From: richard at python.org (Richard Jones) Date: Sun, 19 May 2013 09:00:14 +1000 Subject: [Distutils] 504 timeouts on pypi.python.org In-Reply-To: <5197EE5E.4080900@palladion.com> References: <5197EE5E.4080900@palladion.com> Message-ID: Hi Tres, Noah noticed some resource issues yesterday and killed an offending process. That could have been the cause of the problem. Richard On 19 May 2013 07:10, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 05/18/2013 03:16 PM, Donald Stufft wrote: > > > > On May 17, 2013, at 6:29 PM, Tres Seaver > > wrote: > > > >> Both in the web interface and in registration / upload of releases. > >> > > > Is this still happening to you? > > This afternoon I was able to register / upload the release for which I > noticed the problem yesterday. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlGX7l4ACgkQ+gerLs4ltQ6hLgCgw0eK/PNjmAWz12+LZqnLQfJt > TgYAoIzreHKL8WxXAzNMEyyNBuknC7h/ > =djJM > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun May 19 02:50:04 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 18 May 2013 20:50:04 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 Message-ID: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> Phase 1 of PEP 438 (http://www.python.org/dev/peps/pep-0438/) has begun. Deployed to production PyPI is: - New packages default to pypi-explicit - Old packages default to pypi-scrape-crawl - Package Maintainers can select which hosting mode to use - Package Maintainers can control what urls show up on their /simple/ index What's still happening: - All existing packages will be processed to determine if they host files on PyPI, if they host files externally but link directly, or if they Host files externally and require scraping the home or download url pages. - Taking the data obtained from processing email users to tell them if their package can be moved to a more restrictive download option (e.g. all their versions are installable directly from PyPI, or from a direct link from PyPI). - A Month after that actually moving them to their detected new hosting mode (if possible on a per project basis). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun May 19 02:51:51 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 18 May 2013 20:51:51 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> Message-ID: <45F907FC-5CAF-4F0A-8CC9-685FC736786A@stufft.io> On May 18, 2013, at 8:50 PM, Donald Stufft wrote: > Phase 1 of PEP 438 (http://www.python.org/dev/peps/pep-0438/) has begun. > > Deployed to production PyPI is: > - New packages default to pypi-explicit > - Old packages default to pypi-scrape-crawl > - Package Maintainers can select which hosting mode to use > - Package Maintainers can control what urls show up on their /simple/ index > > What's still happening: > > - All existing packages will be processed to determine if they host files on PyPI, if they host files externally but link directly, or if they Host files externally and require scraping the home or download url pages. > - Taking the data obtained from processing email users to tell them if their package can be moved to a more restrictive download option (e.g. all their versions are installable directly from PyPI, or from a direct link from PyPI). > - A Month after that actually moving them to their detected new hosting mode (if possible on a per project basis). > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Forgot to mention, both of those options are available by clicking on "urls" when viewing a package you have permissions on, see: http://d.stufft.io/image/2h073q2L3Z29 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun May 19 06:00:45 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 19 May 2013 00:00:45 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <45F907FC-5CAF-4F0A-8CC9-685FC736786A@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <45F907FC-5CAF-4F0A-8CC9-685FC736786A@stufft.io> Message-ID: On May 18, 2013, at 8:51 PM, Donald Stufft wrote: > On May 18, 2013, at 8:50 PM, Donald Stufft wrote: > >> Phase 1 of PEP 438 (http://www.python.org/dev/peps/pep-0438/) has begun. >> >> Deployed to production PyPI is: >> - New packages default to pypi-explicit >> - Old packages default to pypi-scrape-crawl >> - Package Maintainers can select which hosting mode to use >> - Package Maintainers can control what urls show up on their /simple/ index >> >> What's still happening: >> >> - All existing packages will be processed to determine if they host files on PyPI, if they host files externally but link directly, or if they Host files externally and require scraping the home or download url pages. >> - Taking the data obtained from processing email users to tell them if their package can be moved to a more restrictive download option (e.g. all their versions are installable directly from PyPI, or from a direct link from PyPI). >> - A Month after that actually moving them to their detected new hosting mode (if possible on a per project basis). >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > Forgot to mention, both of those options are available by clicking on "urls" when viewing a package you have permissions on, see: http://d.stufft.io/image/2h073q2L3Z29 > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Made a little app to see how many urls need to be scraped in order to install any particular package (or all the packages for a user): https://pypi.python.org/pypi/pypi-show-urls It also shows how many packages are not available from PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From r1chardj0n3s at gmail.com Sun May 19 07:58:01 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Sun, 19 May 2013 15:58:01 +1000 Subject: [Distutils] PEP 438 (Transitioning to release-file hosting on PyPI ) is Accepted Message-ID: Thanks everyone who's been involved in the formulation and discussion of PEP 438 "Transitioning to release-file hosting on PyPI". I have today accepted the PEP and the first phase of implementation is now underway, spearheaded by Donald Stufft. In a related note, I welcome and thank the fourth administration volunteer for PyPI, Donald Stufft! :-) Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun May 19 10:12:06 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 19 May 2013 04:12:06 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <45F907FC-5CAF-4F0A-8CC9-685FC736786A@stufft.io> Message-ID: <57148C7C-863C-4F68-9303-2BEF9E9CB17F@stufft.io> On May 19, 2013, at 12:00 AM, Donald Stufft wrote: > > On May 18, 2013, at 8:51 PM, Donald Stufft wrote: > >> On May 18, 2013, at 8:50 PM, Donald Stufft wrote: >> >>> Phase 1 of PEP 438 (http://www.python.org/dev/peps/pep-0438/) has begun. >>> >>> Deployed to production PyPI is: >>> - New packages default to pypi-explicit >>> - Old packages default to pypi-scrape-crawl >>> - Package Maintainers can select which hosting mode to use >>> - Package Maintainers can control what urls show up on their /simple/ index >>> >>> What's still happening: >>> >>> - All existing packages will be processed to determine if they host files on PyPI, if they host files externally but link directly, or if they Host files externally and require scraping the home or download url pages. >>> - Taking the data obtained from processing email users to tell them if their package can be moved to a more restrictive download option (e.g. all their versions are installable directly from PyPI, or from a direct link from PyPI). >>> - A Month after that actually moving them to their detected new hosting mode (if possible on a per project basis). >>> >>> ----------------- >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig >> >> Forgot to mention, both of those options are available by clicking on "urls" when viewing a package you have permissions on, see: http://d.stufft.io/image/2h073q2L3Z29 >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > Made a little app to see how many urls need to be scraped in order to install any particular package (or all the packages for a user): https://pypi.python.org/pypi/pypi-show-urls > > It also shows how many packages are not available from PyPI. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig And one more app: http://pypi-externals.caremad.io/ Also shows per package: http://pypi-externals.caremad.io/setuptools/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Sun May 19 10:58:30 2013 From: holger at merlinux.eu (holger krekel) Date: Sun, 19 May 2013 08:58:30 +0000 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> Message-ID: <20130519085830.GF30215@merlinux.eu> On Sat, May 18, 2013 at 20:50 -0400, Donald Stufft wrote: > Phase 1 of PEP 438 (http://www.python.org/dev/peps/pep-0438/) has begun. > > Deployed to production PyPI is: > - New packages default to pypi-explicit > - Old packages default to pypi-scrape-crawl > - Package Maintainers can select which hosting mode to use > - Package Maintainers can control what urls show up on their /simple/ index Many thanks, Donald, for your great work on this and also to Richard, for getting this live! Just switched pytest, execnet, pytest-xdist, tox and some more projects and it worked fine! Tensed to see how many people will switch without the mails, probably also depends on everybody making a bit of buzz :) Just quickly blogged a about it: http://holgerkrekel.net/2013/05/19/pep438-is-live-speed-up-python-package-installs-now/ . cheers, holger > What's still happening: > > - All existing packages will be processed to determine if they host files on PyPI, if they host files externally but link directly, or if they Host files externally and require scraping the home or download url pages. > - Taking the data obtained from processing email users to tell them if their package can be moved to a more restrictive download option (e.g. all their versions are installable directly from PyPI, or from a direct link from PyPI). > - A Month after that actually moving them to their detected new hosting mode (if possible on a per project basis). > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From tseaver at palladion.com Sun May 19 16:46:49 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 19 May 2013 10:46:49 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/18/2013 08:50 PM, Donald Stufft wrote: > Phase 1 of PEP 438 (http://www.python.org/dev/peps/pep-0438/) has > begun. > > Deployed to production PyPI is: - New packages default to > pypi-explicit - Old packages default to pypi-scrape-crawl - Package > Maintainers can select which hosting mode to use - Package Maintainers > can control what urls show up on their /simple/ index > > What's still happening: > > - All existing packages will be processed to determine if they host > files on PyPI, if they host files externally but link directly, or if > they Host files externally and require scraping the home or download > url pages. - Taking the data obtained from processing email users to > tell them if their package can be moved to a more restrictive download > option (e.g. all their versions are installable directly from PyPI, or > from a direct link from PyPI). - A Month after that actually moving > them to their detected new hosting mode (if possible on a per project > basis). I would be glad to update all my packages to explicit mode, but would prefer to be able to do that in a single batch (clicking through to 250 or so to set it will be tedious). It would be good to automate removing all external URLs previously sniffed from the long-description, too. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGY5dkACgkQ+gerLs4ltQ5q7ACeOz96JWSdd7+cC8LNTlU7YzpF mIEAmwVURbJGsoYQPGBns+1XK4lY3bwN =/Uyr -----END PGP SIGNATURE----- From donald at stufft.io Sun May 19 16:49:07 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 19 May 2013 10:49:07 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> Message-ID: On May 19, 2013, at 10:46 AM, Tres Seaver wrote: > Signed PGP part > On 05/18/2013 08:50 PM, Donald Stufft wrote: > > Phase 1 of PEP 438 (http://www.python.org/dev/peps/pep-0438/) has > > begun. > > > > Deployed to production PyPI is: - New packages default to > > pypi-explicit - Old packages default to pypi-scrape-crawl - Package > > Maintainers can select which hosting mode to use - Package Maintainers > > can control what urls show up on their /simple/ index > > > > What's still happening: > > > > - All existing packages will be processed to determine if they host > > files on PyPI, if they host files externally but link directly, or if > > they Host files externally and require scraping the home or download > > url pages. - Taking the data obtained from processing email users to > > tell them if their package can be moved to a more restrictive download > > option (e.g. all their versions are installable directly from PyPI, or > > from a direct link from PyPI). - A Month after that actually moving > > them to their detected new hosting mode (if possible on a per project > > basis). > > I would be glad to update all my packages to explicit mode, but would > prefer to be able to do that in a single batch (clicking through to 250 > or so to set it will be tedious). It would be good to automate removing > all external URLs previously sniffed from the long-description, too. Yea, as soon as I figure out how to embed arbitrary javascript into a zope.pagetemplate there will be a toggle checked for all thing. > > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From hqzhou at nju.edu.cn Sun May 19 10:12:21 2013 From: hqzhou at nju.edu.cn (Huiqun Zhou) Date: Sun, 19 May 2013 16:12:21 +0800 Subject: [Distutils] easy_install failed to install numpy Message-ID: Hi, I'm trying to install numpy, but got the following error message. What's wrong? Huiqun Zhou =================================================================== [root at dinosaur ~]# easy_install numpy Searching for numpy Reading http://pypi.python.org/simple/numpy/ Reading http://numpy.scipy.org Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Reading http://numeric.scipy.org Reading http://www.numpy.org Best match: numpy 1.7.1 Downloading http://pypi.python.org/packages/source/n/numpy/numpy-1.7.1.zip#md5=9a72db3cad7a6286c0d22ee43ad9bc6c Processing numpy-1.7.1.zip Running numpy-1.7.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gYo5zx/numpy-1.7.1/egg-dist-tmp-t0XrdT Running from numpy source directory. error: SandboxViolation: open('/dev/null', 'w') {} The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted. This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available. From tseaver at palladion.com Sun May 19 22:08:08 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 19 May 2013 16:08:08 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/19/2013 10:49 AM, Donald Stufft wrote: >>> I would be glad to update all my packages to explicit mode, but >>> would prefer to be able to do that in a single batch (clicking >>> through to 250 or so to set it will be tedious). It would be good >>> to automate removing all external URLs previously sniffed from the >>> long-description, too. > Yea, as soon as I figure out how to embed arbitrary javascript into a > zope.pagetemplate there will be a toggle checked for all thing. You can inline Javascript into a ' at which point other scripts can use 'foo'. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGZMSgACgkQ+gerLs4ltQ6MfQCeMtOF8H2T8AA6wbKmVAWayf8x 4vsAn0ArlbIoCDEZkHgHM1UhFaCr65Sy =Hrdn -----END PGP SIGNATURE----- From tseaver at palladion.com Sun May 19 22:11:03 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 19 May 2013 16:11:03 -0400 Subject: [Distutils] easy_install failed to install numpy In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/19/2013 04:12 AM, Huiqun Zhou wrote: > I'm trying to install numpy, but got the following error message. > What's wrong? NumPy and SciPy aren't setuptools-compatible. You will need to follow the directions on the SciPy site: http://www.scipy.org/Installing_SciPy Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGZMdcACgkQ+gerLs4ltQ6/sgCgu9oul6B8Rj9TzFxTH7VBVDxI plEAnAr6VcU6VmAofTSYHomPwXoNGh6y =NLjX -----END PGP SIGNATURE----- From donald at stufft.io Sun May 19 22:20:43 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 19 May 2013 16:20:43 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> Message-ID: <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> On May 19, 2013, at 4:08 PM, Tres Seaver wrote: > Signed PGP part > On 05/19/2013 10:49 AM, Donald Stufft wrote: > >>> I would be glad to update all my packages to explicit mode, but > >>> would prefer to be able to do that in a single batch (clicking > >>> through to 250 or so to set it will be tedious). It would be good > >>> to automate removing all external URLs previously sniffed from the > >>> long-description, too. > > > Yea, as soon as I figure out how to embed arbitrary javascript into a > > zope.pagetemplate there will be a toggle checked for all thing. > > You can inline Javascript into a ' > > at which point other scripts can use 'foo'. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Hrm, ZPT doesn't seem to be stripping the CDATA or unescaping the strings? https://gist.github.com/dstufft/5608838 is what i have in the template file and that appears verbatim in the output? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Sun May 19 23:50:37 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 19 May 2013 17:50:37 -0400 Subject: [Distutils] easy_install failed to install numpy In-Reply-To: References: Message-ID: On Sun, May 19, 2013 at 4:12 AM, Huiqun Zhou wrote: > Hi, > > I'm trying to install numpy, but got the following error message. What's wrong? You're using an older version of setuptools or distribute that doesn't support packages writing to /dev/null in their installation scripts. If you are using setuptools (not distribute), you can upgrade using "easy_install -U setuptools" to get the latest development snapshot, which fixes this problem. If you are using distribute, upgrade to a newer version. I can't guarantee you that this will let you install numpy the rest of the way, but it will at least fix the SandboxViolation error. From pje at telecommunity.com Mon May 20 00:09:12 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 19 May 2013 18:09:12 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <20130519085830.GF30215@merlinux.eu> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: On Sun, May 19, 2013 at 4:58 AM, holger krekel wrote: > Tensed to see how many people will switch without the mails I'll be waiting for the mails, myself, on account of I'm hoping it'll give me a nice list of which of my packages I can switch, so as not to need to go check on all of them myself. It occurred to me today, something I probably should have mentioned earlier in the PEP process, but it'd be nice to be able to switch off home page links without switching off download links. I never have anything to spider in my homepage links, only ever in the download links. Ah well, it'll probably be moot soon. If I understand the PEP correctly, I should be able to write a script that posts #md5-tagged links to my development snapshots, so I can include those directly instead of using directory spidering from the download URLs. Maybe I'll write an "uplink" command extension for setuptools to generate and send the links to PyPI, using the same hooks the "upload" command does. Then I could tie it straight into my build process for development snapshots, without having to run a separate script. From richard at python.org Mon May 20 01:26:48 2013 From: richard at python.org (Richard Jones) Date: Mon, 20 May 2013 09:26:48 +1000 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: On 20 May 2013 08:09, PJ Eby wrote: > On Sun, May 19, 2013 at 4:58 AM, holger krekel wrote: > > Tensed to see how many people will switch without the mails > > I'll be waiting for the mails, myself, on account of I'm hoping it'll > give me a nice list of which of my packages I can switch, so as not to > need to go check on all of them myself. > Donald wrote a handy script to help make this easier: https://pypi.python.org/pypi/pypi-show-urls Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 20 03:02:22 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 19 May 2013 21:02:22 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: <065EA407-96A8-4988-A063-781B5C1373CC@stufft.io> On May 19, 2013, at 7:26 PM, Richard Jones wrote: > On 20 May 2013 08:09, PJ Eby wrote: > On Sun, May 19, 2013 at 4:58 AM, holger krekel wrote: > > Tensed to see how many people will switch without the mails > > I'll be waiting for the mails, myself, on account of I'm hoping it'll > give me a nice list of which of my packages I can switch, so as not to > need to go check on all of them myself. > > Donald wrote a handy script to help make this easier: > > https://pypi.python.org/pypi/pypi-show-urls > > > Richard > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig The emails will still go out of course, it just takes awhile to spider all of PyPI :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Mon May 20 08:18:10 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 20 May 2013 08:18:10 +0200 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> Message-ID: On Sun, May 19, 2013 at 10:20 PM, Donald Stufft wrote: > Hrm, ZPT doesn't seem to be stripping the CDATA or unescaping the strings? > > https://gist.github.com/dstufft/5608838 is what i have in the template file and that appears verbatim in the output? Yes? It will escape *data* inserted into the template (unless told not to), but what is in the template will appear in the output unescaped. I'm not sure how any template system can work otherwise, but perhaps I've been using Zope too long. :-) //Lennart From donald at stufft.io Mon May 20 08:21:05 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 May 2013 02:21:05 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> Message-ID: <1E1E4662-B82B-40BD-861B-6D36813A65EE@stufft.io> On May 20, 2013, at 2:18 AM, Lennart Regebro wrote: > On Sun, May 19, 2013 at 10:20 PM, Donald Stufft wrote: >> Hrm, ZPT doesn't seem to be stripping the CDATA or unescaping the strings? >> >> https://gist.github.com/dstufft/5608838 is what i have in the template file and that appears verbatim in the output? > > Yes? It will escape *data* inserted into the template (unless told not > to), but what is in the template will appear in the output unescaped. > I'm not sure how any template system can work otherwise, but perhaps > I've been using Zope too long. :-) > > //Lennart Maybe you can tell me what I'm doing wrong? I need to insert a Does this not work for you? I'm currently looking at a Zope3 app that does precisely this in its working page templates. > I need to insert a but I haven't seen one in a really long time. > He also said to just put the javascript in the body of the script but > xml escape it. Which I did, and when the template was rendered the > data was still xml escaped and again invalid javascript. I think scripts in XHTML were supposed to be XML-escaped. AFAIU zope.pagetemplate was designed back when XHTML was supposed to be The Bright Future of the Web. Marius Gedminas -- Always proofread carefully to see if you any words out. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From regebro at gmail.com Mon May 20 08:59:00 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 20 May 2013 08:59:00 +0200 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <1E1E4662-B82B-40BD-861B-6D36813A65EE@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> <1E1E4662-B82B-40BD-861B-6D36813A65EE@stufft.io> Message-ID: On Mon, May 20, 2013 at 8:21 AM, Donald Stufft wrote: > He also said to just put the javascript in the body of the script but xml escape it. Which I did, and when the template was rendered the data was still xml escaped and again invalid javascript. I think there is a misunderstanding. When you are putting it into the template itself, you typically don't need to do anything. You just add it to the template. //Lennart From donald at stufft.io Mon May 20 13:14:06 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 May 2013 07:14:06 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <20130520064402.GA3069@fridge.pov.lt> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> <1E1E4662-B82B-40BD-861B-6D36813A65EE@stufft.io> <20130520064402.GA3069@fridge.pov.lt> Message-ID: On May 20, 2013, at 2:44 AM, Marius Gedminas wrote: > On Mon, May 20, 2013 at 02:21:05AM -0400, Donald Stufft wrote: >> >> On May 20, 2013, at 2:18 AM, Lennart Regebro wrote: >> >>> On Sun, May 19, 2013 at 10:20 PM, Donald Stufft wrote: >>>> Hrm, ZPT doesn't seem to be stripping the CDATA or unescaping the strings? >>>> >>>> https://gist.github.com/dstufft/5608838 is what i have in the template file and that appears verbatim in the output? >>> >>> Yes? It will escape *data* inserted into the template (unless told not >>> to), but what is in the template will appear in the output unescaped. >>> I'm not sure how any template system can work otherwise, but perhaps >>> I've been using Zope too long. :-) >>> >>> //Lennart >> >> Maybe you can tell me what I'm doing wrong? > > Using zope.pagetemplate. ;) > > More seriously, zope.pagetemplate has two parsing modes: HTML and XML. > Nobody actually uses the XML mode (pt files start with an > declaration, all tal/metal namespaces must be explicitly defined using > xmlns:tal=url-that-nobody-can-remember). The HTML mode allows you to > write Javascript just like you would do it in a browser, with no extra > XML-quoting: > > > > Does this not work for you? I'm currently looking at a Zope3 app that > does precisely this in its working page templates. Nope, copy/pasted that directly and this is what I got on page load (render): https://gist.github.com/dstufft/5611660 I think PyPI might be using XML mode because I seem to recall there being a comment or a commit message referencing the need to do something a particular way because we were in XML mode. > >> I need to insert a > > but I haven't seen one in a really long time. > >> He also said to just put the javascript in the body of the script but >> xml escape it. Which I did, and when the template was rendered the >> data was still xml escaped and again invalid javascript. > > I think scripts in XHTML were supposed to be XML-escaped. AFAIU > zope.pagetemplate was designed back when XHTML was supposed to be The > Bright Future of the Web. > > Marius Gedminas > -- > Always proofread carefully to see if you any words out. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 20 13:16:18 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 May 2013 07:16:18 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <20130520064402.GA3069@fridge.pov.lt> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> <1E1E4662-B82B-40BD-861B-6D36813A65EE@stufft.io> <20130520064402.GA3069@fridge.pov.lt> Message-ID: <55C14645-C52E-4DAE-8E93-BF10C30F07DB@stufft.io> On May 20, 2013, at 2:44 AM, Marius Gedminas wrote: > On Mon, May 20, 2013 at 02:21:05AM -0400, Donald Stufft wrote: >> >> On May 20, 2013, at 2:18 AM, Lennart Regebro wrote: >> >>> On Sun, May 19, 2013 at 10:20 PM, Donald Stufft wrote: >>>> Hrm, ZPT doesn't seem to be stripping the CDATA or unescaping the strings? >>>> >>>> https://gist.github.com/dstufft/5608838 is what i have in the template file and that appears verbatim in the output? >>> >>> Yes? It will escape *data* inserted into the template (unless told not >>> to), but what is in the template will appear in the output unescaped. >>> I'm not sure how any template system can work otherwise, but perhaps >>> I've been using Zope too long. :-) >>> >>> //Lennart >> >> Maybe you can tell me what I'm doing wrong? > > Using zope.pagetemplate. ;) > > More seriously, zope.pagetemplate has two parsing modes: HTML and XML. > Nobody actually uses the XML mode (pt files start with an > declaration, all tal/metal namespaces must be explicitly defined using > xmlns:tal=url-that-nobody-can-remember). The HTML mode allows you to > write Javascript just like you would do it in a browser, with no extra > XML-quoting: > > > > Does this not work for you? I'm currently looking at a Zope3 app that > does precisely this in its working page templates. > >> I need to insert a > > but I haven't seen one in a really long time. Using this works though, so awesome! > >> He also said to just put the javascript in the body of the script but >> xml escape it. Which I did, and when the template was rendered the >> data was still xml escaped and again invalid javascript. > > I think scripts in XHTML were supposed to be XML-escaped. AFAIU > zope.pagetemplate was designed back when XHTML was supposed to be The > Bright Future of the Web. > > Marius Gedminas > -- > Always proofread carefully to see if you any words out. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 20 13:25:37 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 May 2013 07:25:37 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <55C14645-C52E-4DAE-8E93-BF10C30F07DB@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <9E212BC9-6205-42A9-B519-52A5E976A2D0@stufft.io> <1E1E4662-B82B-40BD-861B-6D36813A65EE@stufft.io> <20130520064402.GA3069@fridge.pov.lt> <55C14645-C52E-4DAE-8E93-BF10C30F07DB@stufft.io> Message-ID: <9822D453-7D6D-4E06-924C-F900684E31BA@stufft.io> And the urls page now has a toggle all checkbox. Thanks again! On May 20, 2013, at 7:16 AM, Donald Stufft wrote: > > On May 20, 2013, at 2:44 AM, Marius Gedminas wrote: > >> On Mon, May 20, 2013 at 02:21:05AM -0400, Donald Stufft wrote: >>> >>> On May 20, 2013, at 2:18 AM, Lennart Regebro wrote: >>> >>>> On Sun, May 19, 2013 at 10:20 PM, Donald Stufft wrote: >>>>> Hrm, ZPT doesn't seem to be stripping the CDATA or unescaping the strings? >>>>> >>>>> https://gist.github.com/dstufft/5608838 is what i have in the template file and that appears verbatim in the output? >>>> >>>> Yes? It will escape *data* inserted into the template (unless told not >>>> to), but what is in the template will appear in the output unescaped. >>>> I'm not sure how any template system can work otherwise, but perhaps >>>> I've been using Zope too long. :-) >>>> >>>> //Lennart >>> >>> Maybe you can tell me what I'm doing wrong? >> >> Using zope.pagetemplate. ;) >> >> More seriously, zope.pagetemplate has two parsing modes: HTML and XML. >> Nobody actually uses the XML mode (pt files start with an >> declaration, all tal/metal namespaces must be explicitly defined using >> xmlns:tal=url-that-nobody-can-remember). The HTML mode allows you to >> write Javascript just like you would do it in a browser, with no extra >> XML-quoting: >> >> >> >> Does this not work for you? I'm currently looking at a Zope3 app that >> does precisely this in its working page templates. >> >>> I need to insert a >> >> but I haven't seen one in a really long time. > > Using this works though, so awesome! > >> >>> He also said to just put the javascript in the body of the script but >>> xml escape it. Which I did, and when the template was rendered the >>> data was still xml escaped and again invalid javascript. >> >> I think scripts in XHTML were supposed to be XML-escaped. AFAIU >> zope.pagetemplate was designed back when XHTML was supposed to be The >> Bright Future of the Web. >> >> Marius Gedminas >> -- >> Always proofread carefully to see if you any words out. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 20 14:18:40 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 May 2013 08:18:40 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: On May 19, 2013, at 6:09 PM, PJ Eby wrote: > On Sun, May 19, 2013 at 4:58 AM, holger krekel wrote: >> Tensed to see how many people will switch without the mails > > I'll be waiting for the mails, myself, on account of I'm hoping it'll > give me a nice list of which of my packages I can switch, so as not to > need to go check on all of them myself. > > It occurred to me today, something I probably should have mentioned > earlier in the PEP process, but it'd be nice to be able to switch off > home page links without switching off download links. I never have > anything to spider in my homepage links, only ever in the download > links. > > Ah well, it'll probably be moot soon. If I understand the PEP > correctly, I should be able to write a script that posts #md5-tagged > links to my development snapshots, so I can include those directly > instead of using directory spidering from the download URLs. Maybe > I'll write an "uplink" command extension for setuptools to generate > and send the links to PyPI, using the same hooks the "upload" command > does. Then I could tie it straight into my build process for > development snapshots, without having to run a separate script. Sorry I forgot to reply to this. Yes this is completely accurate as far as what you'll be able to do (and can do right now). Emails will be coming soon I hope, my processing of PyPI is in the o's now. Just a data point to show the difference this change can have on the large scale, I'm 3 days into processing all of PyPI looking for things that can be downloaded and isntalled and I'm only on the o's. For kicks I did the same thing but with spidering external sites turned off and it took about 10 minutes to process all of PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ronaldoussoren at mac.com Mon May 20 14:30:51 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 20 May 2013 14:30:51 +0200 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: <45F907FC-5CAF-4F0A-8CC9-685FC736786A@stufft.io> References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <45F907FC-5CAF-4F0A-8CC9-685FC736786A@stufft.io> Message-ID: <0109D218-6099-4865-8846-AA0CB802261F@mac.com> On 19 May, 2013, at 2:51, Donald Stufft wrote: >> > > Forgot to mention, both of those options are available by clicking on "urls" when viewing a package you have permissions on, see: http://d.stufft.io/image/2h073q2L3Z29 I get a "Forbidden" error when following the "urls" link in Safari on OSX 10.8. The link does work properly with Chrome on the same machine. Ronald From pje at telecommunity.com Mon May 20 18:05:01 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 20 May 2013 12:05:01 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: On Sun, May 19, 2013 at 7:26 PM, Richard Jones wrote: > Donald wrote a handy script to help make this easier: > > https://pypi.python.org/pypi/pypi-show-urls Doesn't seem to work for me: $ pypi-show-urls -u pje Traceback (most recent call last): File "/usr/bin/pypi-show-urls", line 8, in load_entry_point('pypi-show-urls==2.1.1', 'console_scripts', 'pypi-show-urls')() File "build/bdist.linux-i686/egg/pkg_resources.py", line 318, in load_entry_point File "build/bdist.linux-i686/egg/pkg_resources.py", line 2221, in load_entry_point File "build/bdist.linux-i686/egg/pkg_resources.py", line 1954, in load File "build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py", line 24, in ImportError: No module named pip.req From regebro at gmail.com Mon May 20 19:03:22 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 20 May 2013 19:03:22 +0200 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: On Mon, May 20, 2013 at 6:05 PM, PJ Eby wrote: > On Sun, May 19, 2013 at 7:26 PM, Richard Jones wrote: >> Donald wrote a handy script to help make this easier: >> >> https://pypi.python.org/pypi/pypi-show-urls > > Doesn't seem to work for me: > > $ pypi-show-urls -u pje > Traceback (most recent call last): > File "/usr/bin/pypi-show-urls", line 8, in > load_entry_point('pypi-show-urls==2.1.1', 'console_scripts', > 'pypi-show-urls')() > File "build/bdist.linux-i686/egg/pkg_resources.py", line 318, in > load_entry_point > File "build/bdist.linux-i686/egg/pkg_resources.py", line 2221, in > load_entry_point > File "build/bdist.linux-i686/egg/pkg_resources.py", line 1954, in load > File "build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py", > line 24, in > ImportError: No module named pip.req Do you have pip installed? //Lennart From donald at stufft.io Mon May 20 20:12:37 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 May 2013 14:12:37 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: <5103C81C-8696-4C33-89E4-0881549E5D1B@stufft.io> It requires pip because it process requirements files too. I don't think pip works in install_requires though. A few versions older didn't parse requirements files and didn't have pip requirement. On May 20, 2013, at 12:05 PM, PJ Eby wrote: > On Sun, May 19, 2013 at 7:26 PM, Richard Jones wrote: >> Donald wrote a handy script to help make this easier: >> >> https://pypi.python.org/pypi/pypi-show-urls > > Doesn't seem to work for me: > > $ pypi-show-urls -u pje > Traceback (most recent call last): > File "/usr/bin/pypi-show-urls", line 8, in > load_entry_point('pypi-show-urls==2.1.1', 'console_scripts', > 'pypi-show-urls')() > File "build/bdist.linux-i686/egg/pkg_resources.py", line 318, in > load_entry_point > File "build/bdist.linux-i686/egg/pkg_resources.py", line 2221, in > load_entry_point > File "build/bdist.linux-i686/egg/pkg_resources.py", line 1954, in load > File "build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py", > line 24, in > ImportError: No module named pip.req > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From pje at telecommunity.com Mon May 20 21:20:13 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 20 May 2013 15:20:13 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: On Mon, May 20, 2013 at 1:03 PM, Lennart Regebro wrote: > On Mon, May 20, 2013 at 6:05 PM, PJ Eby wrote: >> On Sun, May 19, 2013 at 7:26 PM, Richard Jones wrote: >>> Donald wrote a handy script to help make this easier: >>> >>> https://pypi.python.org/pypi/pypi-show-urls >> >> Doesn't seem to work for me: >> >> $ pypi-show-urls -u pje >> Traceback (most recent call last): >> File "/usr/bin/pypi-show-urls", line 8, in >> load_entry_point('pypi-show-urls==2.1.1', 'console_scripts', >> 'pypi-show-urls')() >> File "build/bdist.linux-i686/egg/pkg_resources.py", line 318, in >> load_entry_point >> File "build/bdist.linux-i686/egg/pkg_resources.py", line 2221, in >> load_entry_point >> File "build/bdist.linux-i686/egg/pkg_resources.py", line 1954, in load >> File "build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py", >> line 24, in >> ImportError: No module named pip.req > > Do you have pip installed? No, but installing it didn't help; I got an ElementTree exception next: $ pypi-show-urls -u pje Download candidates for PEAK ============================ Traceback (most recent call last): File "/usr/bin/pypi-show-urls", line 8, in load_entry_point('pypi-show-urls==2.1.1', 'console_scripts', 'pypi-show-urls')() File "build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py", line 148, in main File "/usr/lib/python2.6/xml/etree/ElementPath.py", line 198, in findall return _compile(path).findall(element) File "/usr/lib/python2.6/xml/etree/ElementPath.py", line 176, in _compile p = Path(path) File "/usr/lib/python2.6/xml/etree/ElementPath.py", line 93, in __init__ "expected path separator (%s)" % (op or tag) SyntaxError: expected path separator ([) (And of course the package should specify that it has an install-time requirement for pip.) From donald at stufft.io Mon May 20 21:29:16 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 May 2013 15:29:16 -0400 Subject: [Distutils] PEP 438 - Transition Phase 1 In-Reply-To: References: <5AC97ECC-712E-4CB3-98C1-C5C99C74B086@stufft.io> <20130519085830.GF30215@merlinux.eu> Message-ID: Someone else got that error on 2.6 they said 2.7 worked for them. I wasn't able to reproduce on either 2.6 or 2.7. I can add the dep info when I get home. Currently sitting in the ER for my wife. On May 20, 2013, at 3:20 PM, PJ Eby wrote: > On Mon, May 20, 2013 at 1:03 PM, Lennart Regebro wrote: >> On Mon, May 20, 2013 at 6:05 PM, PJ Eby wrote: >>> On Sun, May 19, 2013 at 7:26 PM, Richard Jones wrote: >>>> Donald wrote a handy script to help make this easier: >>>> >>>> https://pypi.python.org/pypi/pypi-show-urls >>> >>> Doesn't seem to work for me: >>> >>> $ pypi-show-urls -u pje >>> Traceback (most recent call last): >>> File "/usr/bin/pypi-show-urls", line 8, in >>> load_entry_point('pypi-show-urls==2.1.1', 'console_scripts', >>> 'pypi-show-urls')() >>> File "build/bdist.linux-i686/egg/pkg_resources.py", line 318, in >>> load_entry_point >>> File "build/bdist.linux-i686/egg/pkg_resources.py", line 2221, in >>> load_entry_point >>> File "build/bdist.linux-i686/egg/pkg_resources.py", line 1954, in load >>> File "build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py", >>> line 24, in >>> ImportError: No module named pip.req >> >> Do you have pip installed? > > No, but installing it didn't help; I got an ElementTree exception next: > > $ pypi-show-urls -u pje > > Download candidates for PEAK > ============================ > Traceback (most recent call last): > File "/usr/bin/pypi-show-urls", line 8, in > load_entry_point('pypi-show-urls==2.1.1', 'console_scripts', > 'pypi-show-urls')() > File "build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py", > line 148, in main > File "/usr/lib/python2.6/xml/etree/ElementPath.py", line 198, in findall > return _compile(path).findall(element) > File "/usr/lib/python2.6/xml/etree/ElementPath.py", line 176, in _compile > p = Path(path) > File "/usr/lib/python2.6/xml/etree/ElementPath.py", line 93, in __init__ > "expected path separator (%s)" % (op or tag) > SyntaxError: expected path separator ([) > > (And of course the package should specify that it has an install-time > requirement for pip.) > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From catalog-sig-bounces at python.org Wed May 22 13:53:58 2013 From: catalog-sig-bounces at python.org (catalog-sig-bounces at python.org) Date: Wed, 22 May 2013 13:53:58 +0200 Subject: [Distutils] Forward of moderated message Message-ID: An embedded message was scrubbed... From: Alexandr Romanov Subject: Re: Error while trying to do a fresh mirror sync Date: Wed, 22 May 2013 11:19:03 +0000 (UTC) Size: 4281 URL: From alex at romanov.ws Wed May 22 18:11:13 2013 From: alex at romanov.ws (Alexandr Romanov) Date: Wed, 22 May 2013 20:11:13 +0400 Subject: [Distutils] Error while trying to do a fresh mirror sync Message-ID: <1DC3024E-F537-4602-B5B5-822B5D2539F9@romanov.ws> Hi, i'm trying to make a new mirror and get following error: ---- ~ # pep381run /var/www/pypi1/ Synchronizing iterator Copying /packages/source/i/iterator/iterator-1.1.0.zip Copying /packages/source/i/iterator/iterator-1.2.0.zip Traceback (most recent call last): File "/usr/local/bin/pep381run", line 30, in state.synchronize() File "/usr/local/lib/python2.6/dist-packages/pep381client/__init__.py", line 119, in synchronize self._synchronize() File "/usr/local/lib/python2.6/dist-packages/pep381client/__init__.py", line 159, in _synchronize self.maybe_copy_file(project, file) File "/usr/local/lib/python2.6/dist-packages/pep381client/__init__.py", line 237, in maybe_copy_file r = h.getresponse() File "/usr/lib/python2.6/httplib.py", line 980, in getresponse raise ResponseNotReady() httplib.ResponseNotReady ---- I already read this : http://mail.python.org/pipermail/catalog-sig/2013-February/005224.html now it looks similar - in http://pypi.python.org/packages/source/i/iterator/ there is no files at all :-( From tk47 at students.poly.edu Wed May 22 18:20:13 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Wed, 22 May 2013 12:20:13 -0400 Subject: [Distutils] Error while trying to do a fresh mirror sync In-Reply-To: <1DC3024E-F537-4602-B5B5-822B5D2539F9@romanov.ws> References: <1DC3024E-F537-4602-B5B5-822B5D2539F9@romanov.ws> Message-ID: <519CF03D.3070803@students.poly.edu> On Wed May 22 12:11:13 2013, Alexandr Romanov wrote: > > i'm trying to make a new mirror and get following error: > Have you tried bandersnatch? It is so much nicer than pep381run. https://bitbucket.org/ctheune/bandersnatch From maphew at gmail.com Thu May 23 02:12:31 2013 From: maphew at gmail.com (Matt Wilkie) Date: Wed, 22 May 2013 17:12:31 -0700 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? Message-ID: Hi folks, *I've been up one side of the internet and down the other, and haven't seen this question discussed before, but as has often been the case perhaps it's just my search-fu which is weak, in which case, please kindly direct me to the answer or documentation where I can find it. Thanks :)* How to distribute a usable python program to a python which doesn't have Distribute installed? I've created a Windows Binary installer using python setup.py bdist_wininst. The setup.py contains a console script entry point, so that a Windows .exe file is created and placed in %python%\Scripts on the destination machine, as per Automatic Script Creation . However running the installed script on a machine with a virgin python install yields: D:\Py3.2.5> scripts\foo.exeTraceback (most recent call last): File "D:\Py3.2.5\scripts\foo-script.py", line 5, in from pkg_resources import load_entry_pointImportError: No module named pkg_resources No module named pkg_resourcestells me this error is because Distribute is not installed. How do I get my installer to include Distribute so I don't have to tell our users *"before you install our program you have to go install this other program"*? thanks in advance for your time, -matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu May 23 07:22:56 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 23 May 2013 01:22:56 -0400 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: On Wed, May 22, 2013 at 8:12 PM, Matt Wilkie wrote: > How do I get my installer to include Distribute so I don't have to tell our > users "before you install our program you have to go install this other > program"? Setuptools (which Distribute is based on) is designed for shipping libraries, not applications; it's developer installer, not primarily an end-user installer for applications. So you probably should be using py2exe instead. Alternatively, you can bundle a copy of pkg_resources.py, or use a script that doesn't depend on entry points, or copy your script to "foo-script.py" alongside the .exe launcher, and manually include those two files as scripts (not using entry points) in your setup() definition. Any of these approaches will solve the problem; it's mostly a matter of your preferences or other requirements. From donald at stufft.io Thu May 23 07:37:28 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 23 May 2013 01:37:28 -0400 Subject: [Distutils] New Restrictions on Package Names Message-ID: <15209BF4-F8A7-410B-855C-C41F7E38C129@stufft.io> Hello! I've just deployed a change to PyPI that restricts all *new* packages names to the follow: * Must start with an ASCII alphanumeric (``[a-zA-Z0-9]``) * Must contain only: - ASCII letters (``[a-zA-Z]``) - ASCII digits (``[0-9]``) - underscores (``_``) - hyphens (``-``) - periods (``.``) * Must end with an ASCII alphanumeric (``[a-zA-Z0-9]``) Projects that have already registered with a name that doesn't follow these restrictions will still be able to upload. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Thu May 23 08:07:37 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 23 May 2013 08:07:37 +0200 Subject: [Distutils] New Restrictions on Package Names In-Reply-To: <15209BF4-F8A7-410B-855C-C41F7E38C129@stufft.io> References: <15209BF4-F8A7-410B-855C-C41F7E38C129@stufft.io> Message-ID: On Thu, May 23, 2013 at 7:37 AM, Donald Stufft wrote: > Hello! > > I've just deployed a change to PyPI that restricts all *new* packages names > to the follow: > > * Must start with an ASCII alphanumeric (``[a-zA-Z0-9]``) > * Must contain only: > - ASCII letters (``[a-zA-Z]``) > - ASCII digits (``[0-9]``) > - underscores (``_``) > - hyphens (``-``) > - periods (``.``) > * Must end with an ASCII alphanumeric (``[a-zA-Z0-9]``) > > Projects that have already registered with a name that doesn't follow these > restrictions will still be able to upload. This explains the new release of Unicode snowman. From donald at stufft.io Thu May 23 12:57:03 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 23 May 2013 06:57:03 -0400 Subject: [Distutils] New Restrictions on Package Names In-Reply-To: References: <15209BF4-F8A7-410B-855C-C41F7E38C129@stufft.io> Message-ID: <12791487-F4C9-4DE6-9B99-41BA44A12C6F@stufft.io> On May 23, 2013, at 2:07 AM, Lennart Regebro wrote: > On Thu, May 23, 2013 at 7:37 AM, Donald Stufft wrote: >> Hello! >> >> I've just deployed a change to PyPI that restricts all *new* packages names >> to the follow: >> >> * Must start with an ASCII alphanumeric (``[a-zA-Z0-9]``) >> * Must contain only: >> - ASCII letters (``[a-zA-Z]``) >> - ASCII digits (``[0-9]``) >> - underscores (``_``) >> - hyphens (``-``) >> - periods (``.``) >> * Must end with an ASCII alphanumeric (``[a-zA-Z0-9]``) >> >> Projects that have already registered with a name that doesn't follow these >> restrictions will still be able to upload. > > This explains the new release of Unicode snowman. ;) From holger at merlinux.eu Fri May 24 11:39:56 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 24 May 2013 09:39:56 +0000 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: Message-ID: <20130524093956.GO7789@merlinux.eu> On Sat, May 18, 2013 at 00:16 +0000, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > > > Is there an API for "all the metadata for everything" that doesn't > > require one web request per package version? Maybe something like an > > rdiff-backup of a database? > > Well, the red-dove.com metadata works at three levels: > > http://www.red-dove.com/pypi/projects/projects.json > > gives a list of all PYPI project names, then e.g. > > http://www.red-dove.com/pypi/projects/W/wheel/project.json > > gives all the version numbers for wheel with download URLs, MD5 hashes (and > importantly, dependencies) and e.g. > > http://www.red-dove.com/pypi/projects/W/wheel/package-1.0.0a2.json > > will give all the particulars of that particular wheel version, including > dependencies, sources, build and test options, package data etc. Nice. How do you actually get at the dependencies? Don't you need to execute setup.py for that? best, holger > So, the middle level gives a useful overview of a project sufficient for > dependency resolution (it's what distlib and distil use). > > The data is kept reasonably fresh by periodically looking at what's changed > recently on PyPI and updating the metadata for new releases etc. > > Regards, > > Vinay Sajip > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From vinay_sajip at yahoo.co.uk Fri May 24 13:17:48 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 24 May 2013 12:17:48 +0100 (BST) Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <20130524093956.GO7789@merlinux.eu> References: <20130524093956.GO7789@merlinux.eu> Message-ID: <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> > From: holger krekel > > Nice.? How do you actually get at the dependencies?? Don't you > need to execute setup.py for that? > Yes, that's how it's done. However, the idea is to do it once per uploaded release and remember the results, so an installer tool like pip doesn't have to download and run setup.py every time :-) Regards, Vinay Sajip From donald at stufft.io Fri May 24 13:20:21 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 24 May 2013 07:20:21 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> Message-ID: <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> On May 24, 2013, at 7:17 AM, Vinay Sajip wrote: >> From: holger krekel > > >> >> Nice. How do you actually get at the dependencies? Don't you >> need to execute setup.py for that? >> > > Yes, that's how it's done. However, the idea is to do it once per uploaded release and remember the results, so an installer tool like pip doesn't have to download and run setup.py every time :-) So what you're saying is I can root your machine with a setup.py? ;) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Fri May 24 15:54:05 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 24 May 2013 13:54:05 +0000 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> Message-ID: <20130524135405.GQ7789@merlinux.eu> On Fri, May 24, 2013 at 07:20 -0400, Donald Stufft wrote: > On May 24, 2013, at 7:17 AM, Vinay Sajip wrote: > > >> From: holger krekel > > > > > >> > >> Nice. How do you actually get at the dependencies? Don't you > >> need to execute setup.py for that? > >> > > > > Yes, that's how it's done. However, the idea is to do it once per uploaded release and remember the results, so an installer tool like pip doesn't have to download and run setup.py every time :-) > > So what you're saying is I can root your machine with a setup.py? ;) That's the immediate risk, indeed :) However, i guess one could use a VM with a chroot and a dedicated user and timeout the setup after 20 seconds or so to regain some safety. It's a bit horrible but OTOH i'd really like to have this information (especially the deps) without requiring everybody to switch to a new packaging format first. holger From donald at stufft.io Fri May 24 15:55:56 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 24 May 2013 09:55:56 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <20130524135405.GQ7789@merlinux.eu> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> Message-ID: <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> On May 24, 2013, at 9:54 AM, holger krekel wrote: > On Fri, May 24, 2013 at 07:20 -0400, Donald Stufft wrote: >> On May 24, 2013, at 7:17 AM, Vinay Sajip wrote: >> >>>> From: holger krekel >>> >>> >>>> >>>> Nice. How do you actually get at the dependencies? Don't you >>>> need to execute setup.py for that? >>>> >>> >>> Yes, that's how it's done. However, the idea is to do it once per uploaded release and remember the results, so an installer tool like pip doesn't have to download and run setup.py every time :-) >> >> So what you're saying is I can root your machine with a setup.py? ;) > > That's the immediate risk, indeed :) However, i guess one could use a VM > with a chroot and a dedicated user and timeout the setup after 20 seconds > or so to regain some safety. It's a bit horrible but OTOH i'd really > like to have this information (especially the deps) without requiring > everybody to switch to a new packaging format first. > > holger Most packages also have an egg-info inside of them you can parse. Of course the issue is that you're only going to get the requirements of the system that ran setup.py, either the authors or the servers. Which doesn't accurately represent all of the dependencies all of the time. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Fri May 24 16:01:16 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 24 May 2013 14:01:16 +0000 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> Message-ID: <20130524140116.GR7789@merlinux.eu> On Fri, May 24, 2013 at 09:55 -0400, Donald Stufft wrote: > On May 24, 2013, at 9:54 AM, holger krekel wrote: > > > On Fri, May 24, 2013 at 07:20 -0400, Donald Stufft wrote: > >> On May 24, 2013, at 7:17 AM, Vinay Sajip wrote: > >> > >>>> From: holger krekel > >>> > >>> > >>>> > >>>> Nice. How do you actually get at the dependencies? Don't you > >>>> need to execute setup.py for that? > >>>> > >>> > >>> Yes, that's how it's done. However, the idea is to do it once per uploaded release and remember the results, so an installer tool like pip doesn't have to download and run setup.py every time :-) > >> > >> So what you're saying is I can root your machine with a setup.py? ;) > > > > That's the immediate risk, indeed :) However, i guess one could use a VM > > with a chroot and a dedicated user and timeout the setup after 20 seconds > > or so to regain some safety. It's a bit horrible but OTOH i'd really > > like to have this information (especially the deps) without requiring > > everybody to switch to a new packaging format first. > > > > holger > > Most packages also have an egg-info inside of them you can parse. > > Of course the issue is that you're only going to get the requirements of the system that ran setup.py, either the authors or the servers. Which doesn't accurately represent all of the dependencies all of the time. True but maybe it would go a long way for most packages. I need dep information mostly for finding out in which dep configuration a package's tests ran successfully. I guess just installing it into a fresh env and "pip freeze" it with some platform info might more directly suit my needs. best, holger > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From marius at pov.lt Fri May 24 16:51:55 2013 From: marius at pov.lt (Marius Gedminas) Date: Fri, 24 May 2013 17:51:55 +0300 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <20130524140116.GR7789@merlinux.eu> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <20130524140116.GR7789@merlinux.eu> Message-ID: <20130524145155.GA30969@fridge.pov.lt> On Fri, May 24, 2013 at 02:01:16PM +0000, holger krekel wrote: > On Fri, May 24, 2013 at 09:55 -0400, Donald Stufft wrote: > > Most packages also have an egg-info inside of them you can parse. > > > > Of course the issue is that you're only going to get the > > requirements of the system that ran setup.py, either the authors or > > the servers. Which doesn't accurately represent all of the > > dependencies all of the time. > > True but maybe it would go a long way for most packages. In that case you might find https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py useful. Marius Gedminas -- HOST SYSTEM RESPONDING, PROBABLY UP... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From vinay_sajip at yahoo.co.uk Fri May 24 17:21:23 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 24 May 2013 15:21:23 +0000 (UTC) Subject: [Distutils] =?utf-8?q?does_pypi_or_red-dove_have_a_better_firehos?= =?utf-8?q?e_API=09than_=22download_all_the_packages=22=3F?= References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > Most packages also have an egg-info inside of them you can parse. I don't know how accurate that information is - IIRC pip always runs egg-info on downloaded archives. I presume this is to get the correct dependencies which are relevant to the installation system. The red-dove metadata is not always completely accurate since it depends on the specific environment the metadata scan runs in (most issues come from dependencies which differ according to Python version, but there could be OS-specific dependencies too which I'm not capturing). I view this automatically generated metadata as a stop-gap until better metadata comes along; I needed to experiment with metadata formats and migration approaches with real data. No doubt there will be tools to help migrate setup.py metadata to alternative metadata formats as my code currently does, by which I mean tools that users can run on their own machines so I can get some sleep at night :-) Regards, Vinay Sajip From donald at stufft.io Fri May 24 17:32:09 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 24 May 2013 11:32:09 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> Message-ID: <27026F04-DC09-4F11-AAA5-F3F2302F2D64@stufft.io> On May 24, 2013, at 11:21 AM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> Most packages also have an egg-info inside of them you can parse. > > I don't know how accurate that information is - IIRC pip always runs > egg-info on downloaded archives. I presume this is to get the correct > dependencies which are relevant to the installation system. That information is typically accurate but it's for the system that generated the package. > > The red-dove metadata is not always completely accurate since it depends on > the specific environment the metadata scan runs in (most issues come from > dependencies which differ according to Python version, but there could be > OS-specific dependencies too which I'm not capturing). I view this > automatically generated metadata as a stop-gap until better metadata comes > along; I needed to experiment with metadata formats and migration approaches > with real data. No doubt there will be tools to help migrate setup.py > metadata to alternative metadata formats as my code currently does, by which > I mean tools that users can run on their own machines so I can get some > sleep at night :-) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Fri May 24 17:35:17 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 24 May 2013 15:35:17 +0000 (UTC) Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <20130524140116.GR7789@merlinux.eu> Message-ID: holger krekel merlinux.eu> writes: > True but maybe it would go a long way for most packages. I need dep > information mostly for finding out in which dep configuration a > package's tests ran successfully. I guess just installing it into a > fresh env and "pip freeze" it with some platform info might more > directly suit my needs. If you're talking about doing this with untrusted packages, then from a security perspective, I presume Donald could root your machine too ;-) Regards, Vinay Sajip From dholth at gmail.com Fri May 24 18:06:48 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 24 May 2013 12:06:48 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <20130524145155.GA30969@fridge.pov.lt> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <20130524140116.GR7789@merlinux.eu> <20130524145155.GA30969@fridge.pov.lt> Message-ID: On Fri, May 24, 2013 at 10:51 AM, Marius Gedminas wrote: > On Fri, May 24, 2013 at 02:01:16PM +0000, holger krekel wrote: >> On Fri, May 24, 2013 at 09:55 -0400, Donald Stufft wrote: >> > Most packages also have an egg-info inside of them you can parse. >> > >> > Of course the issue is that you're only going to get the >> > requirements of the system that ran setup.py, either the authors or >> > the servers. Which doesn't accurately represent all of the >> > dependencies all of the time. >> >> True but maybe it would go a long way for most packages. > > In that case you might find > https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py > useful. > > Marius Gedminas Amazing. http://zope3.pov.lt/py3/ Is anyone interested in doing one for every package on pypi? +1 on potentially inaccurate but useful dependency information. Even the non-regenerated requires.txt from the sdist probably gets you 80-90% there and you can always make sure it was correct during the actual install. You could also parse setup.py to discover which ones definitely don't feature conditional dependencies, marking the served up metadata as "suspect" or "trustworthy". From mal at egenix.com Fri May 24 18:14:25 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 24 May 2013 18:14:25 +0200 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> Message-ID: <519F91E1.2030901@egenix.com> On 24.05.2013 17:21, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> Most packages also have an egg-info inside of them you can parse. > > I don't know how accurate that information is - IIRC pip always runs > egg-info on downloaded archives. I presume this is to get the correct > dependencies which are relevant to the installation system. It would be nice to have the PKG-INFO readily available on the /simple/ index pages. This contains the Requires header as well. At the moment, PKG-INFO is only available on the PyPI edit pages for packages you own, even though access is possible without login: https://pypi.python.org/pypi?name=egenix-mx-base&version=3.2.6&:action=display_pkginfo -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 24 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 38 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Fri May 24 18:18:36 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 24 May 2013 12:18:36 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <519F91E1.2030901@egenix.com> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <519F91E1.2030901@egenix.com> Message-ID: <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> On May 24, 2013, at 12:14 PM, "M.-A. Lemburg" wrote: > On 24.05.2013 17:21, Vinay Sajip wrote: >> Donald Stufft stufft.io> writes: >> >>> Most packages also have an egg-info inside of them you can parse. >> >> I don't know how accurate that information is - IIRC pip always runs >> egg-info on downloaded archives. I presume this is to get the correct >> dependencies which are relevant to the installation system. > > It would be nice to have the PKG-INFO readily available on the /simple/ > index pages. > > This contains the Requires header as well. The requires headers in the PKG-INFO are practically worthless. > > At the moment, PKG-INFO is only available on the PyPI edit pages > for packages you own, even though access is possible without > login: > > https://pypi.python.org/pypi?name=egenix-mx-base&version=3.2.6&:action=display_pkginfo > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 24 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-07-01: EuroPython 2013, Florence, Italy ... 38 days to go > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Fri May 24 18:31:28 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 24 May 2013 18:31:28 +0200 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <519F91E1.2030901@egenix.com> <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> Message-ID: <519F95E0.6030909@egenix.com> On 24.05.2013 18:18, Donald Stufft wrote: > > On May 24, 2013, at 12:14 PM, "M.-A. Lemburg" wrote: > >> On 24.05.2013 17:21, Vinay Sajip wrote: >>> Donald Stufft stufft.io> writes: >>> >>>> Most packages also have an egg-info inside of them you can parse. >>> >>> I don't know how accurate that information is - IIRC pip always runs >>> egg-info on downloaded archives. I presume this is to get the correct >>> dependencies which are relevant to the installation system. >> >> It would be nice to have the PKG-INFO readily available on the /simple/ >> index pages. >> >> This contains the Requires header as well. > > The requires headers in the PKG-INFO are practically worthless. Hmm, looking at a few PKG-INFO entries for Zope packages, I guess you're right. Looks like setuptools forgets to add the header to the PKG-INFO. >> At the moment, PKG-INFO is only available on the PyPI edit pages >> for packages you own, even though access is possible without >> login: >> >> https://pypi.python.org/pypi?name=egenix-mx-base&version=3.2.6&:action=display_pkginfo It does appear in our PKG-INFO entries: https://pypi.python.org/pypi?name=egenix-mxodbc&version=3.2.3&:action=display_pkginfo so I guess setuptools isn't behaving quite right in this respect. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 24 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 38 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Fri May 24 18:33:14 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 24 May 2013 12:33:14 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <519F91E1.2030901@egenix.com> <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> Message-ID: On Fri, May 24, 2013 at 12:18 PM, Donald Stufft wrote: > > On May 24, 2013, at 12:14 PM, "M.-A. Lemburg" wrote: > > On 24.05.2013 17:21, Vinay Sajip wrote: > > Donald Stufft stufft.io> writes: > > Most packages also have an egg-info inside of them you can parse. > > > I don't know how accurate that information is - IIRC pip always runs > egg-info on downloaded archives. I presume this is to get the correct > dependencies which are relevant to the installation system. > > > It would be nice to have the PKG-INFO readily available on the /simple/ > index pages. > > This contains the Requires header as well. > > > The requires headers in the PKG-INFO are practically worthless. Although PKG-INFO specifies a Requires: header and later a Requires-Dist: header, almost no packages use it. The useful requirements information is currently in *.egg-info/requires.txt and requires.txt can't be represented accurately as Requires: headers. The new Metadata 2.0 specification will remedy that by representing everything in a blob of JSON. From donald at stufft.io Fri May 24 18:33:17 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 24 May 2013 12:33:17 -0400 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <519F95E0.6030909@egenix.com> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <519F91E1.2030901@egenix.com> <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> <519F95E0.6030909@egenix.com> Message-ID: On May 24, 2013, at 12:31 PM, "M.-A. Lemburg" wrote: > On 24.05.2013 18:18, Donald Stufft wrote: >> >> On May 24, 2013, at 12:14 PM, "M.-A. Lemburg" wrote: >> >>> On 24.05.2013 17:21, Vinay Sajip wrote: >>>> Donald Stufft stufft.io> writes: >>>> >>>>> Most packages also have an egg-info inside of them you can parse. >>>> >>>> I don't know how accurate that information is - IIRC pip always runs >>>> egg-info on downloaded archives. I presume this is to get the correct >>>> dependencies which are relevant to the installation system. >>> >>> It would be nice to have the PKG-INFO readily available on the /simple/ >>> index pages. >>> >>> This contains the Requires header as well. >> >> The requires headers in the PKG-INFO are practically worthless. > > Hmm, looking at a few PKG-INFO entries for Zope packages, I guess you're > right. Looks like setuptools forgets to add the header to the PKG-INFO. > >>> At the moment, PKG-INFO is only available on the PyPI edit pages >>> for packages you own, even though access is possible without >>> login: >>> >>> https://pypi.python.org/pypi?name=egenix-mx-base&version=3.2.6&:action=display_pkginfo > > It does appear in our PKG-INFO entries: > > https://pypi.python.org/pypi?name=egenix-mxodbc&version=3.2.3&:action=display_pkginfo > > so I guess setuptools isn't behaving quite right in this respect. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 24 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2013-07-01: EuroPython 2013, Florence, Italy ... 38 days to go > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ setuptools is behaving correctly. the Requires field in the old metadata is for requiring python modules (e.g. things you can directly import). Whereas install_requires is for requiring python distributions (e.g. something you can download from PyPI). The old metadata has no provisions for python distributions. That's why the Requires field is "practically worthless". ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Fri May 24 19:05:43 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 24 May 2013 19:05:43 +0200 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <519F91E1.2030901@egenix.com> <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> <519F95E0.6030909@egenix.com> Message-ID: <519F9DE7.2090109@egenix.com> On 24.05.2013 18:33, Donald Stufft wrote: > > On May 24, 2013, at 12:31 PM, "M.-A. Lemburg" wrote: > >> On 24.05.2013 18:18, Donald Stufft wrote: >>> >>> On May 24, 2013, at 12:14 PM, "M.-A. Lemburg" wrote: >>> >>>> On 24.05.2013 17:21, Vinay Sajip wrote: >>>>> Donald Stufft stufft.io> writes: >>>>> >>>>>> Most packages also have an egg-info inside of them you can parse. >>>>> >>>>> I don't know how accurate that information is - IIRC pip always runs >>>>> egg-info on downloaded archives. I presume this is to get the correct >>>>> dependencies which are relevant to the installation system. >>>> >>>> It would be nice to have the PKG-INFO readily available on the /simple/ >>>> index pages. >>>> >>>> This contains the Requires header as well. >>> >>> The requires headers in the PKG-INFO are practically worthless. >> >> Hmm, looking at a few PKG-INFO entries for Zope packages, I guess you're >> right. Looks like setuptools forgets to add the header to the PKG-INFO. >> >>>> At the moment, PKG-INFO is only available on the PyPI edit pages >>>> for packages you own, even though access is possible without >>>> login: >>>> >>>> https://pypi.python.org/pypi?name=egenix-mx-base&version=3.2.6&:action=display_pkginfo >> >> It does appear in our PKG-INFO entries: >> >> https://pypi.python.org/pypi?name=egenix-mxodbc&version=3.2.3&:action=display_pkginfo >> >> so I guess setuptools isn't behaving quite right in this respect. > > setuptools is behaving correctly. the Requires field in the old metadata is for requiring python modules (e.g. things you can directly import). Whereas install_requires is for requiring python distributions (e.g. something you can download from PyPI). The old metadata has no provisions for python distributions. > > That's why the Requires field is "practically worthless". That's an unfortunate interpretation, since the original intention was to describe package dependencies with those fields, but I can see where this is coming from: http://www.python.org/dev/peps/pep-0314/ It mentions the module import names, which often does not correspond to the PyPI package name. PEP 345 clarifies this (by using a new field), but unfortunately never made its was into distutils. Too bad. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 24 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 38 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From notifications at travis-ci.org Fri May 24 21:42:34 2013 From: notifications at travis-ci.org (Travis CI) Date: Fri, 24 May 2013 19:42:34 +0000 Subject: [Distutils] [Fixed] jaraco/setuptools#13 (distribute-maint - 8c89c94) Message-ID: <519fc2a995e70_29729898547@e83554d7-7552-45fd-a355-ef7adc78a200.mail> Build Update for jaraco/setuptools ------------------------------------- Build: #13 Status: Fixed Duration: 21 seconds Commit: 8c89c94 (distribute-maint) Author: PJ Eby Message: First round of prepping setuptools for inclusion in Python 2.5: move site.py to setuptools/site-patch.py; reinstate 'python -m easy_install' support; use distutils' "upload" command when running under 2.5. --HG-- branch : distribute extra : source : fbb6c89a74aa47a556a936202d2e50051b067940 extra : histedit_source : 778122fc56bcca6f5a1dbd5a475df11d29028730 View the changeset: https://github.com/jaraco/setuptools/compare/3b9a57a0c80e...8c89c9438e9e View the full build log and details: https://travis-ci.org/jaraco/setuptools/builds/7473494 -- You can configure recipients for build notifications in your .travis.yml file. See http://about.travis-ci.org/docs/user/build-configuration -------------- next part -------------- An HTML attachment was scrubbed... URL: From notifications at travis-ci.org Fri May 24 21:35:26 2013 From: notifications at travis-ci.org (Travis CI) Date: Fri, 24 May 2013 19:35:26 +0000 Subject: [Distutils] [Broken] jaraco/setuptools#12 (distribute-maint - 2272e24) Message-ID: <519fc0fe16b48_21e990a135190@bb2052a9-80b2-4428-973b-f04fb1219435.mail> Build Update for jaraco/setuptools ------------------------------------- Build: #12 Status: Broken Duration: 53 seconds Commit: 2272e24 (distribute-maint) Author: PJ Eby Message: First round of prepping setuptools for inclusion in Python 2.5: move site.py to setuptools/site-patch.py; reinstate 'python -m easy_install' support; use distutils' "upload" command when running under 2.5. --HG-- branch : distribute rename : site.py => setuptools/site-patch.py extra : source : fbb6c89a74aa47a556a936202d2e50051b067940 extra : histedit_source : 2cce1db2df647a17f6ce37a555207354ed7077f7 View the changeset: https://github.com/jaraco/setuptools/compare/3b9a57a0c80e...2272e2494ccf View the full build log and details: https://travis-ci.org/jaraco/setuptools/builds/7473324 -- You can configure recipients for build notifications in your .travis.yml file. See http://about.travis-ci.org/docs/user/build-configuration -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Fri May 24 22:05:45 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 24 May 2013 20:05:45 +0000 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <20130524145155.GA30969@fridge.pov.lt> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <20130524140116.GR7789@merlinux.eu> <20130524145155.GA30969@fridge.pov.lt> Message-ID: <20130524200545.GT7789@merlinux.eu> On Fri, May 24, 2013 at 17:51 +0300, Marius Gedminas wrote: > On Fri, May 24, 2013 at 02:01:16PM +0000, holger krekel wrote: > > On Fri, May 24, 2013 at 09:55 -0400, Donald Stufft wrote: > > > Most packages also have an egg-info inside of them you can parse. > > > > > > Of course the issue is that you're only going to get the > > > requirements of the system that ran setup.py, either the authors or > > > the servers. Which doesn't accurately represent all of the > > > dependencies all of the time. > > > > True but maybe it would go a long way for most packages. > > In that case you might find > https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py > useful. looks interesting but a bit of a strange UI it seems. Why not accept a tar archive? Or am i missing something? holger From marius at pov.lt Sat May 25 09:03:10 2013 From: marius at pov.lt (Marius Gedminas) Date: Sat, 25 May 2013 10:03:10 +0300 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <20130524200545.GT7789@merlinux.eu> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <20130524140116.GR7789@merlinux.eu> <20130524145155.GA30969@fridge.pov.lt> <20130524200545.GT7789@merlinux.eu> Message-ID: <20130525070310.GA7655@fridge.pov.lt> On Fri, May 24, 2013 at 08:05:45PM +0000, holger krekel wrote: > On Fri, May 24, 2013 at 17:51 +0300, Marius Gedminas wrote: > > On Fri, May 24, 2013 at 02:01:16PM +0000, holger krekel wrote: > > > On Fri, May 24, 2013 at 09:55 -0400, Donald Stufft wrote: > > > > Most packages also have an egg-info inside of them you can parse. > > > > > > > > Of course the issue is that you're only going to get the > > > > requirements of the system that ran setup.py, either the authors or > > > > the servers. Which doesn't accurately represent all of the > > > > dependencies all of the time. > > > > > > True but maybe it would go a long way for most packages. > > > > In that case you might find > > https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py > > useful. > > looks interesting but a bit of a strange UI it seems. Why not accept > a tar archive? Or am i missing something? It's part of a pipeline that builds http://zope3.pov.lt/py3/ At that point I don't have a single tar archive; I have a list of package names, versions and sdist URLs. Marius Gedminas -- Photons have energy, and trying to cram too many into too small of a space can cause a black hole to form, which is, needless to say, not a desirable trait for an optical computer. -- http://scottaaronson.com/blog/?p=261#comment-13693 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From maphew at gmail.com Sat May 25 09:46:37 2013 From: maphew at gmail.com (Matt Wilkie) Date: Sat, 25 May 2013 00:46:37 -0700 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: Thanks PJ. I haven't used py2exe as I don't want to redistribute python also, just our program. I'll dig into your other suggestions and see where I end up. -matt On Wed, May 22, 2013 at 10:22 PM, PJ Eby wrote: > > On Wed, May 22, 2013 at 8:12 PM, Matt Wilkie wrote: > > How do I get my installer to include Distribute so I don't have to tell our > > users "before you install our program you have to go install this other > > program"? > > Setuptools (which Distribute is based on) is designed for shipping > libraries, not applications; it's developer installer, not primarily > an end-user installer for applications. So you probably should be > using py2exe instead. > > Alternatively, you can bundle a copy of pkg_resources.py, or use a > script that doesn't depend on entry points, or copy your script to > "foo-script.py" alongside the .exe launcher, and manually include > those two files as scripts (not using entry points) in your setup() > definition. > > Any of these approaches will solve the problem; it's mostly a matter > of your preferences or other requirements. From ncoghlan at gmail.com Sat May 25 10:02:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 May 2013 18:02:22 +1000 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: Hi Matt, if you don't have any C extensions to deal with and can assume Python is already present on the destination system, then a zipfile with a __main__.py file may also be a suitable solution (see PEP 441 for more info about that feature - it's been supported since 2.6, the PEP just proposes some better tools in the standard library for creating them) Cheers, Nick. On Sat, May 25, 2013 at 5:46 PM, Matt Wilkie wrote: > Thanks PJ. > > I haven't used py2exe as I don't want to redistribute python also, > just our program. I'll dig into your other suggestions and see where I > end up. > > -matt > > > On Wed, May 22, 2013 at 10:22 PM, PJ Eby wrote: >> >> On Wed, May 22, 2013 at 8:12 PM, Matt Wilkie wrote: >> > How do I get my installer to include Distribute so I don't have to tell our >> > users "before you install our program you have to go install this other >> > program"? >> >> Setuptools (which Distribute is based on) is designed for shipping >> libraries, not applications; it's developer installer, not primarily >> an end-user installer for applications. So you probably should be >> using py2exe instead. >> >> Alternatively, you can bundle a copy of pkg_resources.py, or use a >> script that doesn't depend on entry points, or copy your script to >> "foo-script.py" alongside the .exe launcher, and manually include >> those two files as scripts (not using entry points) in your setup() >> definition. >> >> Any of these approaches will solve the problem; it's mostly a matter >> of your preferences or other requirements. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From maphew at gmail.com Sat May 25 09:55:34 2013 From: maphew at gmail.com (Matt Wilkie) Date: Sat, 25 May 2013 00:55:34 -0700 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: slapped reply too early, sorry. > Alternatively, you can bundle a copy of pkg_resources.py, I managed to get this to work by adding it as a script in setup.py: #smelly hack, allows foo.exe to run on machines w/out Distribute scripts = [ 'setup/pkg_resources.py', ], this puts it in "PythonXX\Scripts" on the target machine, as I'm sure you know. What happens if the target machine already has Distribute or Setuptools installed, and thus has pkg_resources somewhere under "PythonXX\Lib\site-packages", and likely a different version to boot? I tried putting it in the same tree as our stuff, "foo/external/pkg_resources.py", but that's not found. I'm assuming because it thinks it's name is "foo.external.pkg_resources". -matt On Sat, May 25, 2013 at 12:46 AM, Matt Wilkie wrote: > Thanks PJ. > > I haven't used py2exe as I don't want to redistribute python also, > just our program. I'll dig into your other suggestions and see where I > end up. > > -matt From maphew at gmail.com Sat May 25 10:06:03 2013 From: maphew at gmail.com (Matt Wilkie) Date: Sat, 25 May 2013 01:06:03 -0700 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: >>... or use a script that doesn't depend on entry points not desirable, as we like the .exe entry points creates and don't want to use a batch file (the "are you sure you want to quit?" message on ctrl-c is annoying) >>, or copy your script to >> "foo-script.py" alongside the .exe launcher, and manually include >> those two files as scripts (not using entry points) in your setup() >> definition. Would you please expand on this? The .exe launcher seems to depend on entry points, or I'm doing something wrong (entirely likely). thanks, -matt From maphew at gmail.com Sat May 25 10:16:31 2013 From: maphew at gmail.com (Matt Wilkie) Date: Sat, 25 May 2013 01:16:31 -0700 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: That looks very interesting Nick! Do you know of any small or smallish programs using this I could inspect and learn from? Most of the references I find surround pyinstaller it's specfile format. thanks, -matt On Sat, May 25, 2013 at 1:02 AM, Nick Coghlan wrote: > Hi Matt, if you don't have any C extensions to deal with and can > assume Python is already present on the destination system, then a > zipfile with a __main__.py file may also be a suitable solution (see > PEP 441 for more info about that feature - it's been supported since > 2.6, the PEP just proposes some better tools in the standard library > for creating them) > > Cheers, > Nick. > > > On Sat, May 25, 2013 at 5:46 PM, Matt Wilkie wrote: >> Thanks PJ. >> >> I haven't used py2exe as I don't want to redistribute python also, >> just our program. I'll dig into your other suggestions and see where I >> end up. >> >> -matt >> >> >> On Wed, May 22, 2013 at 10:22 PM, PJ Eby wrote: >>> >>> On Wed, May 22, 2013 at 8:12 PM, Matt Wilkie wrote: >>> > How do I get my installer to include Distribute so I don't have to tell our >>> > users "before you install our program you have to go install this other >>> > program"? >>> >>> Setuptools (which Distribute is based on) is designed for shipping >>> libraries, not applications; it's developer installer, not primarily >>> an end-user installer for applications. So you probably should be >>> using py2exe instead. >>> >>> Alternatively, you can bundle a copy of pkg_resources.py, or use a >>> script that doesn't depend on entry points, or copy your script to >>> "foo-script.py" alongside the .exe launcher, and manually include >>> those two files as scripts (not using entry points) in your setup() >>> definition. >>> >>> Any of these approaches will solve the problem; it's mostly a matter >>> of your preferences or other requirements. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat May 25 10:58:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 May 2013 18:58:44 +1000 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: On Sat, May 25, 2013 at 6:16 PM, Matt Wilkie wrote: > That looks very interesting Nick! > > Do you know of any small or smallish programs using this I could > inspect and learn from? Most of the references I find surround > pyinstaller it's specfile format. The zipfile execution feature was added primarily as a tool for internal distribution within large Python-using organisations, so I'm not aware of any good examples on the public internet (I assume it's seeing some use inside Google, as that's where the idea originated). Internet distribution tends to use either py2exe and its ilk (because the distributor can't assume a Python interpreter on the destination system) or else give users instructions on installing and using pip or easy_install (because they want their instructions to work across at least the main three platforms). Using pip or easy_install to do the installation also has the side effect of getting pkg_resources onto the target system, since some features of that ecosystem (like entry points) require it. Being able to assume the presence of Python on the target system but *not* the presence of pkg_resources is a fairly idiosyncratic case, which is why you haven't been able to find much specific info on handling it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pje at telecommunity.com Sat May 25 15:16:09 2013 From: pje at telecommunity.com (PJ Eby) Date: Sat, 25 May 2013 09:16:09 -0400 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: On Sat, May 25, 2013 at 4:06 AM, Matt Wilkie wrote: >>>... or use a script that doesn't depend on entry points > > not desirable, as we like the .exe entry points creates and don't want > to use a batch file (the "are you sure you want to quit?" message on > ctrl-c is annoying) > >>>, or copy your script to >>> "foo-script.py" alongside the .exe launcher, and manually include >>> those two files as scripts (not using entry points) in your setup() >>> definition. > > Would you please expand on this? The .exe launcher seems to depend on > entry points, or I'm doing something wrong (entirely likely). The foo.exe launcher doesn't use the entry point, all it does is run an adjacent foo-script.py. You can put anything you want in foo-script.py adjacent to that .exe, and it'll run. As for different versions of pkg_resources on sys.path, Python always puts the directory containing the script first. So if there's a pkg_resources.py adjacent to the script, that will take precedence over any other copy. From jaraco at jaraco.com Sat May 25 22:40:45 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sat, 25 May 2013 20:40:45 +0000 Subject: [Distutils] Announcing Setuptools 0.7b3 Message-ID: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> Setuptools and Distribute are now merged, and the new home can be found at https://bitbucket.org/pypa/setuptools. Issues for Distribute are still being maintained at its old home, but releases are now made from the 'distribute' fork on the setuptools repo. Future releases of Distribute (if any) will be maintenance only. The new setuptools, which is designed to be largely compatible both with Distribute and Setuptools can now be downloaded on the 'downloads' page of the PyPA site: https://bitbucket.org/pypa/setuptools/downloads For those familiar with setuptools/distribute, the upgrade process is pretty straightforward: remove your old versions of distribute and/or setuptools and install the latest. To install the latest, extract the .tar.gz, and run setup.py in the target environment. Windows .msi installers have been provided as well. Ez_setup.py will not yet work until the official release is made to PyPI. I encourage all of those with experience or interest in packaging to try out the new setuptools in your environments. Test it against your tool chain. Provide feedback and file issues in the project site. This limited public release is provided as an advance release prior to the full public release. Assuming there are no reported blocking issues, a final 0.7 release will be made to PyPI in the coming weeks. Regards, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From kamal.mustafa at gmail.com Sun May 26 00:08:12 2013 From: kamal.mustafa at gmail.com (Mohd Kamal Bin Mustafa) Date: Sun, 26 May 2013 06:08:12 +0800 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: On Sat, May 25, 2013 at 4:16 PM, Matt Wilkie wrote: > That looks very interesting Nick! > > Do you know of any small or smallish programs using this I could > inspect and learn from? Most of the references I find surround > pyinstaller it's specfile format. This article explain about using the __main__.py file - http://sayspy.blogspot.com/2010/03/various-ways-of-distributing-python.html As for example, one that I recently found - https://github.com/pagekite/PyPagekite/ From tseaver at palladion.com Sun May 26 00:36:38 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 25 May 2013 18:36:38 -0400 Subject: [Distutils] Announcing Setuptools 0.7b3 In-Reply-To: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> References: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: <51A13CF6.6020804@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/25/2013 04:40 PM, Jason R. Coombs wrote: > Setuptools and Distribute are now merged, and the new home can be > found at https://bitbucket.org/pypa/setuptools. > > Issues for Distribute are still being maintained at its old home, but > releases are now made from the 'distribute' fork on the setuptools > repo. Future releases of Distribute (if any) will be maintenance > only. > > The new setuptools, which is designed to be largely compatible both > with Distribute and Setuptools can now be downloaded on the > 'downloads' page of the PyPA site: > > https://bitbucket.org/pypa/setuptools/downloads > > For those familiar with setuptools/distribute, the upgrade process is > pretty straightforward: remove your old versions of distribute and/or > setuptools and install the latest. To install the latest, extract the > .tar.gz, and run setup.py in the target environment. Windows .msi > installers have been provided as well. Ez_setup.py will not yet work > until the official release is made to PyPI. > > I encourage all of those with experience or interest in packaging to > try out the new setuptools in your environments. Test it against your > tool chain. Provide feedback and file issues in the project site. > > This limited public release is provided as an advance release prior to > the full public release. Assuming there are no reported blocking > issues, a final 0.7 release will be made to PyPI in the coming weeks. Congratulations, and thanks to the PyPA developers for coming together to produce this unified release. FWIW, I was just able to update a setuptools 0.6 virtualenv via: $ bin/easy_install -U https://bitbucket.org/pypa/setuptools/downloads/setuptools-0.7b3.tar.gz For a distribute-based virtualenv, I had to delete the 'distribute' entry in 'site-pacakges' after running the 'easy_install' command. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGhPPYACgkQ+gerLs4ltQ5zngCg1iTC2xwLcvZT/MHSi8bNlFDl clgAnjXOCbXqN0rW1ihwWEBUCFzEFC5U =MF9s -----END PGP SIGNATURE----- From mal at egenix.com Sun May 26 00:44:04 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 26 May 2013 00:44:04 +0200 Subject: [Distutils] does pypi or red-dove have a better firehose API than "download all the packages"? In-Reply-To: <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> References: <20130524093956.GO7789@merlinux.eu> <1369394268.35158.YahooMailNeo@web171405.mail.ir2.yahoo.com> <82E5D2CB-7717-4777-B9DE-0F8E4A11EFB1@stufft.io> <20130524135405.GQ7789@merlinux.eu> <67FB5A65-C0AE-442F-B7C0-8FAF6FF42066@stufft.io> <519F91E1.2030901@egenix.com> <7751160D-F523-4740-932F-EAFE5B47E910@stufft.io> Message-ID: <51A13EB4.2020909@egenix.com> On 24.05.2013 18:18, Donald Stufft wrote: > > On May 24, 2013, at 12:14 PM, "M.-A. Lemburg" wrote: > >> On 24.05.2013 17:21, Vinay Sajip wrote: >>> Donald Stufft stufft.io> writes: >>> >>>> Most packages also have an egg-info inside of them you can parse. >>> >>> I don't know how accurate that information is - IIRC pip always runs >>> egg-info on downloaded archives. I presume this is to get the correct >>> dependencies which are relevant to the installation system. >> >> It would be nice to have the PKG-INFO readily available on the /simple/ >> index pages. >> >> This contains the Requires header as well. > > The requires headers in the PKG-INFO are practically worthless. > >> >> At the moment, PKG-INFO is only available on the PyPI edit pages >> for packages you own, even though access is possible without >> login: >> >> https://pypi.python.org/pypi?name=egenix-mx-base&version=3.2.6&:action=display_pkginfo Leaving aside the unusable Requires header, could we still get PKG-INFO exposed in the /simple/ index ? This would then make it possible to access the basic meta-data cached via the CDN and without having to download the package. Something like: 3.2.6 PKG-INFO
for each release. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 26 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 36 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From pje at telecommunity.com Sun May 26 00:44:39 2013 From: pje at telecommunity.com (PJ Eby) Date: Sat, 25 May 2013 18:44:39 -0400 Subject: [Distutils] Announcing Setuptools 0.7b3 In-Reply-To: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> References: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On Sat, May 25, 2013 at 4:40 PM, Jason R. Coombs wrote: > https://bitbucket.org/pypa/setuptools/downloads You may be already aware of this, but the installation instructions don't match the available downloads. I was previously thinking we should just drop binary distributions of setuptools altogether anyway, and make ez_setup.py the standard way to install setuptools on all platforms. (Which would help address things like installation conflicts w/existing Distribute, etc.) From jaraco at jaraco.com Sun May 26 01:02:16 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sat, 25 May 2013 23:02:16 +0000 Subject: [Distutils] Announcing Setuptools 0.7b3 In-Reply-To: References: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: I'll be sure to address that before the final release. I have been tempted to just support sdist installs and follow the technique that Distribute used, except for this one comment I heard in person by a newer Windows user recently, that they still hear that Setuptools is still considered friendlier than Distribute. But if PJE is on board with source installs, now is a good time for that switch, so let's do it. > -----Original Message----- > From: PJ Eby [mailto:pje at telecommunity.com] > Sent: Saturday, 25 May, 2013 18:45 > To: Jason R. Coombs > Cc: distutils-sig at python.org > Subject: Re: [Distutils] Announcing Setuptools 0.7b3 > > On Sat, May 25, 2013 at 4:40 PM, Jason R. Coombs > wrote: > > https://bitbucket.org/pypa/setuptools/downloads > > You may be already aware of this, but the installation instructions don't > match the available downloads. I was previously thinking we should just > drop binary distributions of setuptools altogether anyway, and make > ez_setup.py the standard way to install setuptools on all platforms. (Which > would help address things like installation conflicts w/existing Distribute, > etc.) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From pje at telecommunity.com Sun May 26 01:29:27 2013 From: pje at telecommunity.com (PJ Eby) Date: Sat, 25 May 2013 19:29:27 -0400 Subject: [Distutils] Announcing Setuptools 0.7b3 In-Reply-To: References: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On Sat, May 25, 2013 at 7:02 PM, Jason R. Coombs wrote: > I heard in person by a newer > Windows user recently, that they still hear that Setuptools is still > considered friendlier than Distribute. Without knowing what exactly they considered friendlier, I'm not sure what to suggest. Even if you use a GUI to install setuptools, it's not going to do you much good without getting to a command line! From ncoghlan at gmail.com Sun May 26 02:42:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 May 2013 10:42:07 +1000 Subject: [Distutils] Announcing Setuptools 0.7b3 In-Reply-To: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> References: <022de825792945e78e247c00752c969f@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On Sun, May 26, 2013 at 6:40 AM, Jason R. Coombs wrote: > Setuptools and Distribute are now merged, and the new home can be found at > https://bitbucket.org/pypa/setuptools. Yay! Thanks for the work both you and PJE have put into making this happen :) The pieces of Python's improved packaging story start to come together... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From maphew at gmail.com Sun May 26 06:47:55 2013 From: maphew at gmail.com (Matt Wilkie) Date: Sat, 25 May 2013 21:47:55 -0700 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: > Being able to assume the presence of Python on the target system but > *not* the presence of pkg_resources is a fairly idiosyncratic case, > which is why you haven't been able to find much specific info on > handling it. This might be a Windows thing. I'm coming primarily from that background, where for most "if it doesn't have a GUI installer, it doesn't exist", and Distribute doesn't have an installer. I guess it's idiosyncratic because those coming before me on a similar path would have just gravitated to NSIS or similar for this need. -matt From maphew at gmail.com Sun May 26 07:28:48 2013 From: maphew at gmail.com (Matt Wilkie) Date: Sat, 25 May 2013 22:28:48 -0700 Subject: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed? In-Reply-To: References: Message-ID: > The foo.exe launcher doesn't use the entry point, all it does is run > an adjacent foo-script.py. You can put anything you want in > foo-script.py adjacent to that .exe, and it'll run. Ahhh, perfect! I've verified it works for me, and much better than bundling a duplicate of Distribute. Thanks! [...later]: is a postinstall the best (only?) means to replace the contents of foo-script.py? By experimentation I've learned the `entry_points=` script supersedes a `scripts=` script of the same name. > As for different versions of pkg_resources on sys.path, Python always > puts the directory containing the script first. So if there's a > pkg_resources.py adjacent to the script, that will take precedence > over any other copy. Thanks, this is good to know. best, -matt From noah at coderanger.net Sun May 26 19:34:58 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 26 May 2013 10:34:58 -0700 Subject: [Distutils] Good news everyone, PyPI is behind a CDN Message-ID: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun May 26 19:39:04 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 26 May 2013 13:39:04 -0400 Subject: [Distutils] [Infrastructure] Good news everyone, PyPI is behind a CDN In-Reply-To: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> Message-ID: <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> On May 26, 2013, at 1:34 PM, Noah Kantrowitz wrote: > > > but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. > > --Noah > > ________________________________________________ > Infrastructure mailing list > Infrastructure at python.org > http://mail.python.org/mailman/listinfo/infrastructure > Unsubscribe: http://mail.python.org/mailman/options/infrastructure/donald%40stufft.io One addition. Eventually PyPI will move towards using the Fastly API for cache invalidation so that uploading a new package _will_ be immediately available. But that was deferred in order to get the larger benefit *now*. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jim at zope.com Sun May 26 20:01:45 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 26 May 2013 14:01:45 -0400 Subject: [Distutils] [Infrastructure] Good news everyone, PyPI is behind a CDN In-Reply-To: <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> Message-ID: On Sun, May 26, 2013 at 1:39 PM, Donald Stufft wrote: ... > One addition. Eventually PyPI will move towards using the Fastly API for > cache invalidation so that uploading a new package _will_ be immediately > available. Nice! > But that was deferred in order to get the larger benefit *now*. Thanks for helping to make this happen. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From donald at stufft.io Sun May 26 20:03:21 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 26 May 2013 14:03:21 -0400 Subject: [Distutils] [Infrastructure] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> Message-ID: <67E7C626-F31A-4AD1-92F2-3C8F159794D0@stufft.io> On May 26, 2013, at 2:00 PM, Jim Fulton wrote: > On Sun, May 26, 2013 at 1:39 PM, Donald Stufft wrote: >> >> On May 26, 2013, at 1:34 PM, Noah Kantrowitz wrote: >> >> >> >> but seriously, at long last today it was my honor to throw the DNS switch to >> move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for >> doing much of the heavy lifting on the PyPI side, and to Fastly for >> graciously offering to host us. What does this mean for everyone? Well the >> biggest change is PyPI should get a whole lot faster. There are two major >> downsides however. There will now be a delay of several minutes in some >> cases between updating a package and having it be installable, and download >> counts will now be even more incorrect than they were before. The PyPI >> admins are discussing what to do about download counts long-term, but for >> now we all feel that the performance and availability benefits outweigh the >> loss. If anyone has any questions, or hears anything about issues with PyPI >> please don't hesitate to contact me. > > This is great news! Is this just the downloads? Or does it include > the simple index too? Default Policy is Not to Cache (To handle logged in or dynamic pages since PyPI doesn't have the proper Vary headers). /simple/.* Has a policy for it to cache for 10minutes (If I recall correctly) /packages/.* Has a policy for it to cache for 1 hour (If I recall correctly) So they are both cached, but because we aren't invalidating the cache yet the simple index has a low timeout so there is only a short wait required for a new upload to be available. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jim at zope.com Sun May 26 20:00:25 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 26 May 2013 14:00:25 -0400 Subject: [Distutils] [Infrastructure] Good news everyone, PyPI is behind a CDN In-Reply-To: <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> Message-ID: On Sun, May 26, 2013 at 1:39 PM, Donald Stufft wrote: > > On May 26, 2013, at 1:34 PM, Noah Kantrowitz wrote: > > > > but seriously, at long last today it was my honor to throw the DNS switch to > move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for > doing much of the heavy lifting on the PyPI side, and to Fastly for > graciously offering to host us. What does this mean for everyone? Well the > biggest change is PyPI should get a whole lot faster. There are two major > downsides however. There will now be a delay of several minutes in some > cases between updating a package and having it be installable, and download > counts will now be even more incorrect than they were before. The PyPI > admins are discussing what to do about download counts long-term, but for > now we all feel that the performance and availability benefits outweigh the > loss. If anyone has any questions, or hears anything about issues with PyPI > please don't hesitate to contact me. This is great news! Is this just the downloads? Or does it include the simple index too? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jonas at geiregat.org Sun May 26 22:08:34 2013 From: jonas at geiregat.org (Jonas Geiregat) Date: Sun, 26 May 2013 22:08:34 +0200 Subject: [Distutils] pypi protocol Message-ID: Hello, I'm fairly interested in the internals of pypi. Ultimately pypi is a HTTP interface to a package database. I went looking for a document, blog post or anything describing the communication between pypi client and server. I ended up reading pypiserver's source code to find out the internals. Is there a reason why there haven't been a PEP describing the pypi interface ? Regards, Jonas. From richard at python.org Mon May 27 01:09:37 2013 From: richard at python.org (Richard Jones) Date: Mon, 27 May 2013 09:09:37 +1000 Subject: [Distutils] pypi protocol In-Reply-To: References: Message-ID: The only interface documentation is linked from http://wiki.python.org/moin/CheeseShopDev which includes PEP 301 and http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api The rest is documented-in-code. Richard On 27 May 2013 06:08, Jonas Geiregat wrote: > Hello, > > I'm fairly interested in the internals of pypi. > > Ultimately pypi is a HTTP interface to a package database. > I went looking for a document, blog post or anything describing the > communication between pypi client and server. > > I ended up reading pypiserver's source code to find out the internals. > > Is there a reason why there haven't been a PEP describing the pypi > interface ? > > Regards, > > Jonas. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 27 02:08:35 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 26 May 2013 20:08:35 -0400 Subject: [Distutils] PyPI Download Counts Message-ID: Hello! As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated. There are numerous reasons for their removal/deprecation some of which are: - Technically hard to make work with the new CDN - The CDN is being donated to the PSF, and the donated tier does not offer any form of log access - The work around for not having log access would greatly reduce the utility of the CDN - Highly inaccurate - A number of things prevent the download counts from being inaccurate, some of which include: - pip download cache - Internal or unofficial mirrors - Packages not hosted on PyPI (for comparisons sake) - Mirrors or unofficial grab scripts causing inflated counts (Last I looked 25% of the downloads were from a known mirroring script). - Not particularly useful - Just because a project has been downloaded a lot doesn't mean it's good - Similarly just because a project hasn't been downloaded a lot doesn't mean it's bad In short because it's value is low for various reasons, and the tradeoffs required to make it work are high It has been not an effective use of resources. The API will continue to return values for it in order to not break scripts, however in the future all these values will be set to 0. The Web UI has been modified to no longer display it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Mon May 27 07:19:52 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 27 May 2013 07:19:52 +0200 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> Message-ID: On Sun, May 26, 2013 at 7:34 PM, Noah Kantrowitz wrote: > > > but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. This is going to spell disaster for the coffee industry, as you no longer have to take a coffee break when re-running a buildout. Thanks! //Lennart From holger at merlinux.eu Mon May 27 09:19:29 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 07:19:29 +0000 Subject: [Distutils] pypi protocol In-Reply-To: References: Message-ID: <20130527071929.GW7789@merlinux.eu> Hi Jonas, On Sun, May 26, 2013 at 22:08 +0200, Jonas Geiregat wrote: > Hello, > > I'm fairly interested in the internals of pypi. > > Ultimately pypi is a HTTP interface to a package database. > I went looking for a document, blog post or anything describing the communication between pypi client and server. > > I ended up reading pypiserver's source code to find out the internals. > > Is there a reason why there haven't been a PEP describing the pypi interface ? The protocol pip/easy_install use with pypi.python.org is fairly minimal. Carl and me described it in PEP438 and a number of people reviewed it so it should be rather accurate. See particularly http://www.python.org/dev/peps/pep-0438/#history-and-motivations-for-external-hosting This does not discuss the XMLRPC/json interfaces and ASFAIK there are few pypi implementations apart from the main one who implement them. For implementation of the basic protocol, you can look into pip/index.py or checkout my tested implementation in devpi-server here: https://bitbucket.org/hpk42/devpi-server/src/029db81602bccece0f58be9a66cd67f640ac353a/devpi_server/extpypi.py?at=default#cl-44 cheers, holger From holger at merlinux.eu Mon May 27 09:27:31 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 07:27:31 +0000 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: Message-ID: <20130527072731.GX7789@merlinux.eu> Hi Donald, On Sun, May 26, 2013 at 20:08 -0400, Donald Stufft wrote: > Hello! > > As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated. > > There are numerous reasons for their removal/deprecation some of which are: > - Technically hard to make work with the new CDN > - The CDN is being donated to the PSF, and the donated tier does not offer any form of log access What would be involved money/effort wise to get such access? > - The work around for not having log access would greatly reduce the utility of the CDN > - Highly inaccurate > - A number of things prevent the download counts from being inaccurate, some of which include: > - pip download cache > - Internal or unofficial mirrors > - Packages not hosted on PyPI (for comparisons sake) > - Mirrors or unofficial grab scripts causing inflated counts (Last I looked 25% of the downloads were from a known mirroring script). given the CDN usage of mirrors may drop soon. > - Not particularly useful > - Just because a project has been downloaded a lot doesn't mean it's good > - Similarly just because a project hasn't been downloaded a lot doesn't mean it's bad > > In short because it's value is low for various reasons, and the tradeoffs required to make it work are high It has been not an effective use of resources. > > The API will continue to return values for it in order to not break scripts, however in the future all these values will be set to 0. The Web UI has been modified to no longer display it. While download counts do have the weeknesses you describe they also provide a rough indication of usage which many of us referred to. I used it to determine interest and it partly drove my development efforts. From that angle i am not happy about the change but of course i see the benefits. Not having download counts maybe lets us think harder about better metrics. The number of projects using a package as a dep might be one. cheers, holger > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Mon May 27 09:41:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 May 2013 17:41:17 +1000 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130527072731.GX7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> Message-ID: On Mon, May 27, 2013 at 5:27 PM, holger krekel wrote: > Not having download counts maybe lets us think harder about > better metrics. The number of projects using a package as a dep > might be one. With the current downside being that it's hard for PyPI to figure out that number, too :) Agreed it would be a good number to publish once it's more readily available, too. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Mon May 27 09:59:34 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 07:59:34 +0000 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> Message-ID: <20130527075934.GZ7789@merlinux.eu> On Mon, May 27, 2013 at 17:41 +1000, Nick Coghlan wrote: > On Mon, May 27, 2013 at 5:27 PM, holger krekel wrote: > > Not having download counts maybe lets us think harder about > > better metrics. The number of projects using a package as a dep > > might be one. > > With the current downside being that it's hard for PyPI to figure out > that number, too :) Yip. But something like Vinaj's red-dove approach or Marius' get_deps.py could provide a base. We might think about a docker instance which could allow to quickly spawn new light VMs so we can isolate setup.py runs. (Yes, it's only Linux but it'd be a start). > Agreed it would be a good number to publish once it's more readily > available, too. I think "dep" numbers are mostly interesting for libraries, not so much for applications like django or pyramid or tools like nose/pytest. Another more practical data point would be "does this package even install on win32/linux/osx py26/py27/py33" and even better, do its automated tests pass? If we could evolve to have this info published on pypi.python.org it would be quite useful i think. I am actually currently implementing a system which enables this (the "devpi" system) so i don't mean this all just as "nice to have" theory. I aim to present the status of this work at EuroPython. best, holger > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > From flo at chaoflow.net Mon May 27 10:36:04 2013 From: flo at chaoflow.net (Florian Friesdorf) Date: Mon, 27 May 2013 10:36:04 +0200 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130527075934.GZ7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> <20130527075934.GZ7789@merlinux.eu> Message-ID: <87y5b0u8ej.fsf@eve.Speedport_W_921V_1_21_000> Hi Holger, holger krekel writes: > On Mon, May 27, 2013 at 17:41 +1000, Nick Coghlan wrote: >> On Mon, May 27, 2013 at 5:27 PM, holger krekel wrote: >> > Not having download counts maybe lets us think harder about >> > better metrics. The number of projects using a package as a dep >> > might be one. >> >> With the current downside being that it's hard for PyPI to figure out >> that number, too :) > > Yip. But something like Vinaj's red-dove approach or Marius' get_deps.py > could provide a base. We might think about a docker instance which > could allow to quickly spawn new light VMs so we can isolate setup.py runs. > (Yes, it's only Linux but it'd be a start). nix and nixpkgs allow this isolation on-top off linux, freebsd, OS X and theoretically also cygwin (not sure how good cygwin is supported at the moment). http://nixos.org/nix/ http://nixos.org/nixpkgs/ From nixos.org: Nix is a purely functional package manager. This means that it can ensure that an upgrade to one package cannot break others, that you can always roll back to previous version, that multiple versions of a package can coexist on the same system, and much more. Nixpkgs is a large collection of packages that can be installed with the Nix package manager. >> Agreed it would be a good number to publish once it's more readily >> available, too. > > I think "dep" numbers are mostly interesting for libraries, not so > much for applications like django or pyramid or tools like nose/pytest. > > Another more practical data point would be "does this package even > install on win32/linux/osx py26/py27/py33" and even better, do its automated > tests pass? http://hydra.nixos.org/build/5062796 > If we could evolve to have this info published on pypi.python.org > it would be quite useful i think. I am actually currently implementing > a system which enables this (the "devpi" system) so i don't mean this all just > as "nice to have" theory. I aim to present the status of this work > at EuroPython. Nice! Looking forward to that. If you have any questions about nix/nixpkgs/nixos, especially about the way python packages are packaged, please let me know. Also, it's not set in stone. Personally, I'd love to see hydra.python.org providing builds of all pypi packages and would be happy to help. Also including Domen and Rok for whome I assume the same. You might have other tools that are better suited for you. regards florian -- Florian Friesdorf GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083 Jabber/XMPP: flo at chaoflow.net IRC: chaoflow on freenode,ircnet,blafasel,OFTC -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From noah at coderanger.net Mon May 27 10:55:04 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 27 May 2013 01:55:04 -0700 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130527072731.GX7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> Message-ID: <2B17088E-5E54-42E5-A290-300D1C619BE4@coderanger.net> On May 27, 2013, at 12:27 AM, holger krekel wrote: > Hi Donald, > > On Sun, May 26, 2013 at 20:08 -0400, Donald Stufft wrote: >> Hello! >> >> As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated. >> >> There are numerous reasons for their removal/deprecation some of which are: >> - Technically hard to make work with the new CDN >> - The CDN is being donated to the PSF, and the donated tier does not offer any form of log access > > What would be involved money/effort wise to get such access? > >> - The work around for not having log access would greatly reduce the utility of the CDN >> - Highly inaccurate >> - A number of things prevent the download counts from being inaccurate, some of which include: >> - pip download cache >> - Internal or unofficial mirrors >> - Packages not hosted on PyPI (for comparisons sake) >> - Mirrors or unofficial grab scripts causing inflated counts (Last I looked 25% of the downloads were from a known mirroring script). > > given the CDN usage of mirrors may drop soon. > >> - Not particularly useful >> - Just because a project has been downloaded a lot doesn't mean it's good >> - Similarly just because a project hasn't been downloaded a lot doesn't mean it's bad >> >> In short because it's value is low for various reasons, and the tradeoffs required to make it work are high It has been not an effective use of resources. >> >> The API will continue to return values for it in order to not break scripts, however in the future all these values will be set to 0. The Web UI has been modified to no longer display it. > > While download counts do have the weeknesses you describe they also > provide a rough indication of usage which many of us referred to. > I used it to determine interest and it partly drove my development > efforts. From that angle i am not happy about the change but of course > i see the benefits. > > Not having download counts maybe lets us think harder about > better metrics. The number of projects using a package as a dep > might be one. We do still get some indication of package activity from looking through the logs, it just no longer has a direct correlation. We will see one request hit the backend servers from each shield node per hour when that package is being requested. At some point we could recycle this into some kind of abstract popularity count, but I don't think thats a development priority for anyone right now. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Mon May 27 13:36:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 May 2013 21:36:37 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published Message-ID: After preliminary reviews by Donald and Daniel, I have now pushed the first complete draft of the JSON-based metadata 2.0 proposal to python.org PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ With the rationale and commentary, they're over 3000 lines between them, so I'm not attaching them here. The rationale for many of the changes is at the end of each PEP, along with some comments on features that I have either rejected or deliberately chosen to defer to the next revision of the metadata (at the earliest). Those with BitBucket accounts may also comment inline on the drafts here: PEP 426: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0426.txt?at=default PEP 440: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0440.txt?at=default Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Mon May 27 13:39:40 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 27 May 2013 13:39:40 +0200 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130527075934.GZ7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> <20130527075934.GZ7789@merlinux.eu> Message-ID: On Mon, May 27, 2013 at 9:59 AM, holger krekel wrote: > Another more practical data point would be "does this package even > install on win32/linux/osx py26/py27/py33" and even better, do its automated > tests pass? Those are interesting metrics, but doesn't indicate the packages popularity at all. //Lennart From holger at merlinux.eu Mon May 27 14:08:46 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 12:08:46 +0000 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> Message-ID: <20130527120846.GA7789@merlinux.eu> Hi Noah, Donald, (CC also Richard, Christian), i just checked with a test package and think we might have a cache consistency / changelog API problem. It took me a while but here is the basic thing: I uploaded a test package, changelog API reports it has changed, then i go to its simple page, and some of the time the new release file shows up, sometimes not. Tools like bandersnatch, pep381 and devpi-server (and probably others) use PyPI's changelog API to determine if there are changes. It seems those changes are signalled faster than they become consistently accessible through the CDN. This can lead to inconsistent mirrors because when the CDN has the files there is no change event anymore. Such mirrors are run by companies in-house so i think it's a real problem. Even without mirroring there can be problems because installs are not directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, then 2.0.0 a minute later. I had hoped that a particular ip address sees things consistently. I am not familiar with Fastly's caching properties -- can they notify about the fact that a page/file is consistently up-to-date everywhere? Or can the cache be globally invalidated for a particular page/file? Any other ideas? Failing customizing Fastly usage and also maybe for the short term, is/could there be a special location provided by pypi.python.org which the above tools could use to get at the actual non-cached data? We could then maybe mitigate the problem through updates of the respective tools. That would at least solve the problem for one of my customers i think. best, holger On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: > > > but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. > > --Noah > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From holger at merlinux.eu Mon May 27 14:57:27 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 12:57:27 +0000 Subject: [Distutils] [Infrastructure] Good news everyone, PyPI is behind a CDN In-Reply-To: <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <9C3FD2A9-E178-4194-9984-BF968C432001@stufft.io> Message-ID: <20130527125727.GD7789@merlinux.eu> On Sun, May 26, 2013 at 13:39 -0400, Donald Stufft wrote: > On May 26, 2013, at 1:34 PM, Noah Kantrowitz wrote: > > > > > > > but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. > > > > --Noah > > > > ________________________________________________ > > Infrastructure mailing list > > Infrastructure at python.org > > http://mail.python.org/mailman/listinfo/infrastructure > > Unsubscribe: http://mail.python.org/mailman/options/infrastructure/donald%40stufft.io > > One addition. Eventually PyPI will move towards using the Fastly API for cache invalidation so that uploading a new package _will_ be immediately available. But that was deferred in order to get the larger benefit *now*. Good intention, but it seems to break the changelog api and thus mirroring (see my other post) :/ >From the fastly website it seems simple to invalidate "simple/PROJECT/" when a PROJECT package is registered/uploaded, before triggering a changelog serial update and returning a success code. holger > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From holger at merlinux.eu Mon May 27 15:34:38 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 13:34:38 +0000 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> <20130527075934.GZ7789@merlinux.eu> Message-ID: <20130527133438.GE7789@merlinux.eu> On Mon, May 27, 2013 at 13:39 +0200, Lennart Regebro wrote: > On Mon, May 27, 2013 at 9:59 AM, holger krekel wrote: > > Another more practical data point would be "does this package even > > install on win32/linux/osx py26/py27/py33" and even better, do its automated > > tests pass? > > Those are interesting metrics, but doesn't indicate the packages > popularity at all. right. And to be clear, I wish we would get back download counters even if they are just rough indicators. For me this is a negative aspect of the CDN move as i was using a fully functional mirroring cache -- which currently is broken due to the very move. holger From ronaldoussoren at mac.com Mon May 27 16:44:52 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 27 May 2013 16:44:52 +0200 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: <06FE1104-014B-4EA1-9CAF-6BEC0FAFD4F1@mac.com> On 27 May, 2013, at 13:36, Nick Coghlan wrote: > After preliminary reviews by Donald and Daniel, I have now pushed the > first complete draft of the JSON-based metadata 2.0 proposal to > python.org > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ Could platform_release be added to the set of environment markers? On OSX platform.release() returns a kernel version number, while platform.version() returns a string that cannot easily be used as a marker. It might also be nice to have markers for the marketing name and version of the OS next to the kernel name and release (for example Solaris 8 vs. SunOS 5.8, or darwin 12.3.0 vs. OSX 10.8.3), but that would probably need a PEP of its own to add the functionality to the stdlib before it is used in metadata. BTW. Why can marker expressions use only a limited subset of operators? In particular <, <=, > and >= are not allowed and would be nice to have to specify packages that need a minimal platform version (parts of PyObjC have a dependency on the OSX release because the wrapped library is available in a subset of releases). Listing all OS releases would be fairly useless, platform.release() is too specific to use with '==' as new patch releases of OSX can introduce new micro versions of the kernel. > PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ The versioning spec mentions that distribution tools may refuse to publish distributions that pin the versions of dependencies. I understand why this is needed, and agree in general, but have a usecase that I don't know how to express without pinning. In particular, PyObjC consists of a number of distributions (pyobjc-core, pyobjc-framework-Cocoa, ...) and an umbrella package (pyobjc) what depends on the various distributions to make it easier to install all of PyObjC. The umbrella package currently pins the versions of subpackages to ensure that "pip install pyobjc==2.5.1" installs exactly that version of the entire project. When I'd use the "compatible release" specifier I can no longer easily ensure that users can install an exact version of the entire project, other than by hacking the system: specify a compatible version with an additional level that isn't used by the project (for example ~=2.5.2.0). What is the correct way to create an umberella project without getting yelled at by distribution tools? Ronald > > With the rationale and commentary, they're over 3000 lines between > them, so I'm not attaching them here. > > The rationale for many of the changes is at the end of each PEP, along > with some comments on features that I have either rejected or > deliberately chosen to defer to the next revision of the metadata (at > the earliest). > > Those with BitBucket accounts may also comment inline on the drafts here: > > PEP 426: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0426.txt?at=default > PEP 440: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0440.txt?at=default > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From holger at merlinux.eu Mon May 27 17:46:36 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 15:46:36 +0000 Subject: [Distutils] PyPI Download Counts In-Reply-To: <87y5b0u8ej.fsf@eve.Speedport_W_921V_1_21_000> References: <20130527072731.GX7789@merlinux.eu> <20130527075934.GZ7789@merlinux.eu> <87y5b0u8ej.fsf@eve.Speedport_W_921V_1_21_000> Message-ID: <20130527154636.GF7789@merlinux.eu> Hi Florian, On Mon, May 27, 2013 at 10:36 +0200, Florian Friesdorf wrote: > Hi Holger, > > holger krekel writes: > > On Mon, May 27, 2013 at 17:41 +1000, Nick Coghlan wrote: > >> On Mon, May 27, 2013 at 5:27 PM, holger krekel wrote: > >> > Not having download counts maybe lets us think harder about > >> > better metrics. The number of projects using a package as a dep > >> > might be one. > >> > >> With the current downside being that it's hard for PyPI to figure out > >> that number, too :) > > > > Yip. But something like Vinaj's red-dove approach or Marius' get_deps.py > > could provide a base. We might think about a docker instance which > > could allow to quickly spawn new light VMs so we can isolate setup.py runs. > > (Yes, it's only Linux but it'd be a start). > > nix and nixpkgs allow this isolation on-top off linux, freebsd, OS X and > theoretically also cygwin (not sure how good cygwin is supported at the > moment). > > http://nixos.org/nix/ > http://nixos.org/nixpkgs/ > > From nixos.org: > Nix is a purely functional package manager. This means that it can > ensure that an upgrade to one package cannot break others, that you can > always roll back to previous version, that multiple versions of a > package can coexist on the same system, and much more. > > Nixpkgs is a large collection of packages that can be installed with the > Nix package manager. Interesting stuff, didn't know about it. Did you post this as a suggestion for provisioning an environment to run setup.py (on nix-supported platforms)? If so, i am not sure how it would help exactly. I guess myself i'd aim for a 80% solution for discovering dependencies first. Simplest/quickest wins there :) > >> Agreed it would be a good number to publish once it's more readily > >> available, too. > > > > I think "dep" numbers are mostly interesting for libraries, not so > > much for applications like django or pyramid or tools like nose/pytest. > > > > Another more practical data point would be "does this package even > > install on win32/linux/osx py26/py27/py33" and even better, do its automated > > tests pass? > > http://hydra.nixos.org/build/5062796 > > > If we could evolve to have this info published on pypi.python.org > > it would be quite useful i think. I am actually currently implementing > > a system which enables this (the "devpi" system) so i don't mean this all just > > as "nice to have" theory. I aim to present the status of this work > > at EuroPython. > > Nice! Looking forward to that. > > If you have any questions about nix/nixpkgs/nixos, especially about the > way python packages are packaged, please let me know. Also, it's not set > in stone. are you going to be at EP? It's a long conference and i am more than happy to sit together on this topic for a bit sometimes. best, holger > Personally, I'd love to see hydra.python.org providing builds of all > pypi packages and would be happy to help. Also including Domen and Rok > for whome I assume the same. > > You might have other tools that are better suited for you. > > regards > florian > -- > Florian Friesdorf > GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083 > Jabber/XMPP: flo at chaoflow.net > IRC: chaoflow on freenode,ircnet,blafasel,OFTC From donald at stufft.io Mon May 27 18:39:56 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 May 2013 12:39:56 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <20130527120846.GA7789@merlinux.eu> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> Message-ID: On May 27, 2013, at 8:08 AM, holger krekel wrote: > Hi Noah, Donald, (CC also Richard, Christian), > > i just checked with a test package and think we might have a cache > consistency / changelog API problem. It took me a while but here is > the basic thing: I uploaded a test package, changelog API reports it has > changed, then i go to its simple page, and some of the time the new release > file shows up, sometimes not. > > Tools like bandersnatch, pep381 and devpi-server (and probably others) > use PyPI's changelog API to determine if there are changes. It seems > those changes are signalled faster than they become consistently accessible > through the CDN. This can lead to inconsistent mirrors because when > the CDN has the files there is no change event anymore. Such mirrors > are run by companies in-house so i think it's a real problem. > > Even without mirroring there can be problems because installs are not > directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, > then 2.0.0 a minute later. I had hoped that a particular ip address > sees things consistently. > > I am not familiar with Fastly's caching properties -- can they notify > about the fact that a page/file is consistently up-to-date everywhere? > Or can the cache be globally invalidated for a particular page/file? > Any other ideas? > > Failing customizing Fastly usage and also maybe for the short term, > is/could there be a special location provided by pypi.python.org which > the above tools could use to get at the actual non-cached data? We > could then maybe mitigate the problem through updates of the respective tools. > That would at least solve the problem for one of my customers i think. > > best, > holger > > > On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: >> >> >> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. >> >> --Noah >> > > > >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I mentioned it on twitter but might as well mention it here as well. Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 27 19:50:57 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 May 2013 13:50:57 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> Message-ID: <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> On May 27, 2013, at 12:39 PM, Donald Stufft wrote: > > On May 27, 2013, at 8:08 AM, holger krekel wrote: > >> Hi Noah, Donald, (CC also Richard, Christian), >> >> i just checked with a test package and think we might have a cache >> consistency / changelog API problem. It took me a while but here is >> the basic thing: I uploaded a test package, changelog API reports it has >> changed, then i go to its simple page, and some of the time the new release >> file shows up, sometimes not. >> >> Tools like bandersnatch, pep381 and devpi-server (and probably others) >> use PyPI's changelog API to determine if there are changes. It seems >> those changes are signalled faster than they become consistently accessible >> through the CDN. This can lead to inconsistent mirrors because when >> the CDN has the files there is no change event anymore. Such mirrors >> are run by companies in-house so i think it's a real problem. >> >> Even without mirroring there can be problems because installs are not >> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, >> then 2.0.0 a minute later. I had hoped that a particular ip address >> sees things consistently. >> >> I am not familiar with Fastly's caching properties -- can they notify >> about the fact that a page/file is consistently up-to-date everywhere? >> Or can the cache be globally invalidated for a particular page/file? >> Any other ideas? >> >> Failing customizing Fastly usage and also maybe for the short term, >> is/could there be a special location provided by pypi.python.org which >> the above tools could use to get at the actual non-cached data? We >> could then maybe mitigate the problem through updates of the respective tools. >> That would at least solve the problem for one of my customers i think. >> >> best, >> holger >> >> >> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: >>> >>> >>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. >>> >>> --Noah >>> >> >> >> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > I mentioned it on twitter but might as well mention it here as well. > > Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig /simple/ Pages should now be immediately invalidated when a new package is released. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Mon May 27 20:54:53 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 18:54:53 +0000 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> Message-ID: <20130527185453.GG7789@merlinux.eu> On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote: > On May 27, 2013, at 12:39 PM, Donald Stufft wrote: > > > > > On May 27, 2013, at 8:08 AM, holger krekel wrote: > > > >> Hi Noah, Donald, (CC also Richard, Christian), > >> > >> i just checked with a test package and think we might have a cache > >> consistency / changelog API problem. It took me a while but here is > >> the basic thing: I uploaded a test package, changelog API reports it has > >> changed, then i go to its simple page, and some of the time the new release > >> file shows up, sometimes not. > >> > >> Tools like bandersnatch, pep381 and devpi-server (and probably others) > >> use PyPI's changelog API to determine if there are changes. It seems > >> those changes are signalled faster than they become consistently accessible > >> through the CDN. This can lead to inconsistent mirrors because when > >> the CDN has the files there is no change event anymore. Such mirrors > >> are run by companies in-house so i think it's a real problem. > >> > >> Even without mirroring there can be problems because installs are not > >> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, > >> then 2.0.0 a minute later. I had hoped that a particular ip address > >> sees things consistently. > >> > >> I am not familiar with Fastly's caching properties -- can they notify > >> about the fact that a page/file is consistently up-to-date everywhere? > >> Or can the cache be globally invalidated for a particular page/file? > >> Any other ideas? > >> > >> Failing customizing Fastly usage and also maybe for the short term, > >> is/could there be a special location provided by pypi.python.org which > >> the above tools could use to get at the actual non-cached data? We > >> could then maybe mitigate the problem through updates of the respective tools. > >> That would at least solve the problem for one of my customers i think. > >> > >> best, > >> holger > >> > >> > >> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: > >>> > >>> > >>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. > >>> > >>> --Noah > >>> > >> > >> > >> > >>> _______________________________________________ > >>> Distutils-SIG maillist - Distutils-SIG at python.org > >>> http://mail.python.org/mailman/listinfo/distutils-sig > >> > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> http://mail.python.org/mailman/listinfo/distutils-sig > > > > I mentioned it on twitter but might as well mention it here as well. > > > > Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. > > > > ----------------- > > Donald Stufft > > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > /simple/ Pages should now be immediately invalidated when a new package is released. thanks Donald. Looking at the implementation, i wonder what happens if after ``self._conn.commit()`` a changelog API call arrives, returns changes and a client uses it to retrieve changes before the fastly-purging takes place. It's still a potential race-condition or am i missing something? best, holger > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From donald at stufft.io Mon May 27 20:59:00 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 May 2013 14:59:00 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <20130527185453.GG7789@merlinux.eu> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> Message-ID: On May 27, 2013, at 2:54 PM, holger krekel wrote: > On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote: >> On May 27, 2013, at 12:39 PM, Donald Stufft wrote: >> >>> >>> On May 27, 2013, at 8:08 AM, holger krekel wrote: >>> >>>> Hi Noah, Donald, (CC also Richard, Christian), >>>> >>>> i just checked with a test package and think we might have a cache >>>> consistency / changelog API problem. It took me a while but here is >>>> the basic thing: I uploaded a test package, changelog API reports it has >>>> changed, then i go to its simple page, and some of the time the new release >>>> file shows up, sometimes not. >>>> >>>> Tools like bandersnatch, pep381 and devpi-server (and probably others) >>>> use PyPI's changelog API to determine if there are changes. It seems >>>> those changes are signalled faster than they become consistently accessible >>>> through the CDN. This can lead to inconsistent mirrors because when >>>> the CDN has the files there is no change event anymore. Such mirrors >>>> are run by companies in-house so i think it's a real problem. >>>> >>>> Even without mirroring there can be problems because installs are not >>>> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, >>>> then 2.0.0 a minute later. I had hoped that a particular ip address >>>> sees things consistently. >>>> >>>> I am not familiar with Fastly's caching properties -- can they notify >>>> about the fact that a page/file is consistently up-to-date everywhere? >>>> Or can the cache be globally invalidated for a particular page/file? >>>> Any other ideas? >>>> >>>> Failing customizing Fastly usage and also maybe for the short term, >>>> is/could there be a special location provided by pypi.python.org which >>>> the above tools could use to get at the actual non-cached data? We >>>> could then maybe mitigate the problem through updates of the respective tools. >>>> That would at least solve the problem for one of my customers i think. >>>> >>>> best, >>>> holger >>>> >>>> >>>> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: >>>>> >>>>> >>>>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. >>>>> >>>>> --Noah >>>>> >>>> >>>> >>>> >>>>> _______________________________________________ >>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> http://mail.python.org/mailman/listinfo/distutils-sig >>> >>> I mentioned it on twitter but might as well mention it here as well. >>> >>> Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. >>> >>> ----------------- >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig >> >> >> >> /simple/ Pages should now be immediately invalidated when a new package is released. > > thanks Donald. Looking at the implementation, i wonder what happens if > after ``self._conn.commit()`` a changelog API call arrives, returns changes > and a client uses it to retrieve changes before the fastly-purging takes > place. It's still a potential race-condition or am i missing something? > > best, > holger > >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > > There's no way around a race condition. ``self._conn.commit()`` is what makes the changes available. If we purge prior to committing it then if someone hits the page between the purge and the self._conn.commit() then the client will see a page cached prior to the update (while the change log will appear to be updated). Essentially the same problem we have now. The current implementation does mean that if a client happens to hit between the commit and the purge they'll see old data however that's pretty unlikely. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Mon May 27 21:18:39 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 19:18:39 +0000 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> Message-ID: <20130527191839.GI7789@merlinux.eu> On Mon, May 27, 2013 at 14:59 -0400, Donald Stufft wrote: > On May 27, 2013, at 2:54 PM, holger krekel wrote: > > > On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote: > >> On May 27, 2013, at 12:39 PM, Donald Stufft wrote: > >> > >>> > >>> On May 27, 2013, at 8:08 AM, holger krekel wrote: > >>> > >>>> Hi Noah, Donald, (CC also Richard, Christian), > >>>> > >>>> i just checked with a test package and think we might have a cache > >>>> consistency / changelog API problem. It took me a while but here is > >>>> the basic thing: I uploaded a test package, changelog API reports it has > >>>> changed, then i go to its simple page, and some of the time the new release > >>>> file shows up, sometimes not. > >>>> > >>>> Tools like bandersnatch, pep381 and devpi-server (and probably others) > >>>> use PyPI's changelog API to determine if there are changes. It seems > >>>> those changes are signalled faster than they become consistently accessible > >>>> through the CDN. This can lead to inconsistent mirrors because when > >>>> the CDN has the files there is no change event anymore. Such mirrors > >>>> are run by companies in-house so i think it's a real problem. > >>>> > >>>> Even without mirroring there can be problems because installs are not > >>>> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, > >>>> then 2.0.0 a minute later. I had hoped that a particular ip address > >>>> sees things consistently. > >>>> > >>>> I am not familiar with Fastly's caching properties -- can they notify > >>>> about the fact that a page/file is consistently up-to-date everywhere? > >>>> Or can the cache be globally invalidated for a particular page/file? > >>>> Any other ideas? > >>>> > >>>> Failing customizing Fastly usage and also maybe for the short term, > >>>> is/could there be a special location provided by pypi.python.org which > >>>> the above tools could use to get at the actual non-cached data? We > >>>> could then maybe mitigate the problem through updates of the respective tools. > >>>> That would at least solve the problem for one of my customers i think. > >>>> > >>>> best, > >>>> holger > >>>> > >>>> > >>>> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: > >>>>> > >>>>> > >>>>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. > >>>>> > >>>>> --Noah > >>>>> > >>>> > >>>> > >>>> > >>>>> _______________________________________________ > >>>>> Distutils-SIG maillist - Distutils-SIG at python.org > >>>>> http://mail.python.org/mailman/listinfo/distutils-sig > >>>> > >>>> _______________________________________________ > >>>> Distutils-SIG maillist - Distutils-SIG at python.org > >>>> http://mail.python.org/mailman/listinfo/distutils-sig > >>> > >>> I mentioned it on twitter but might as well mention it here as well. > >>> > >>> Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. > >>> > >>> ----------------- > >>> Donald Stufft > >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > >>> > >>> _______________________________________________ > >>> Distutils-SIG maillist - Distutils-SIG at python.org > >>> http://mail.python.org/mailman/listinfo/distutils-sig > >> > >> > >> > >> /simple/ Pages should now be immediately invalidated when a new package is released. > > > > thanks Donald. Looking at the implementation, i wonder what happens if > > after ``self._conn.commit()`` a changelog API call arrives, returns changes > > and a client uses it to retrieve changes before the fastly-purging takes > > place. It's still a potential race-condition or am i missing something? > > > > best, > > holger > > > >> ----------------- > >> Donald Stufft > >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > >> > > > > > > > There's no way around a race condition. > > ``self._conn.commit()`` is what makes the changes available. If we purge prior to committing it then if someone hits the page between the purge and the self._conn.commit() then the client will see a page cached prior to the update (while the change log will appear to be updated). Essentially the same problem we have now. > > The current implementation does mean that if a client happens to hit between the commit and the purge they'll see old data however that's pretty unlikely. Purging can take a second and also depends on the network connectivity between pypi.python.org and fastly's api to begin with. I am afraid the race-condition is bound to happen and then hard to detect. Not sure how exactly pypi.python.org is deployed but could commit() use a semaphore which also the changelog-APIs use so that the latter only returns after purging (and them some) has happened? I don't think mirrors would mind sometimes waiting a few seconds before the changelog* call returns as long as the state is then consistent. Lastly, i think introducing a bit of internal syncing overhead to commit()/ changelog should be ok because we have only few writes and hardly read load. holger > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From noah at coderanger.net Mon May 27 21:58:10 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 27 May 2013 12:58:10 -0700 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <20130527191839.GI7789@merlinux.eu> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> Message-ID: <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> On May 27, 2013, at 12:18 PM, holger krekel wrote: > On Mon, May 27, 2013 at 14:59 -0400, Donald Stufft wrote: >> On May 27, 2013, at 2:54 PM, holger krekel wrote: >> >>> On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote: >>>> On May 27, 2013, at 12:39 PM, Donald Stufft wrote: >>>> >>>>> >>>>> On May 27, 2013, at 8:08 AM, holger krekel wrote: >>>>> >>>>>> Hi Noah, Donald, (CC also Richard, Christian), >>>>>> >>>>>> i just checked with a test package and think we might have a cache >>>>>> consistency / changelog API problem. It took me a while but here is >>>>>> the basic thing: I uploaded a test package, changelog API reports it has >>>>>> changed, then i go to its simple page, and some of the time the new release >>>>>> file shows up, sometimes not. >>>>>> >>>>>> Tools like bandersnatch, pep381 and devpi-server (and probably others) >>>>>> use PyPI's changelog API to determine if there are changes. It seems >>>>>> those changes are signalled faster than they become consistently accessible >>>>>> through the CDN. This can lead to inconsistent mirrors because when >>>>>> the CDN has the files there is no change event anymore. Such mirrors >>>>>> are run by companies in-house so i think it's a real problem. >>>>>> >>>>>> Even without mirroring there can be problems because installs are not >>>>>> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, >>>>>> then 2.0.0 a minute later. I had hoped that a particular ip address >>>>>> sees things consistently. >>>>>> >>>>>> I am not familiar with Fastly's caching properties -- can they notify >>>>>> about the fact that a page/file is consistently up-to-date everywhere? >>>>>> Or can the cache be globally invalidated for a particular page/file? >>>>>> Any other ideas? >>>>>> >>>>>> Failing customizing Fastly usage and also maybe for the short term, >>>>>> is/could there be a special location provided by pypi.python.org which >>>>>> the above tools could use to get at the actual non-cached data? We >>>>>> could then maybe mitigate the problem through updates of the respective tools. >>>>>> That would at least solve the problem for one of my customers i think. >>>>>> >>>>>> best, >>>>>> holger >>>>>> >>>>>> >>>>>> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: >>>>>>> >>>>>>> >>>>>>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. >>>>>>> >>>>>>> --Noah >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> _______________________________________________ >>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>>> >>>>>> _______________________________________________ >>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>> >>>>> I mentioned it on twitter but might as well mention it here as well. >>>>> >>>>> Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. >>>>> >>>>> ----------------- >>>>> Donald Stufft >>>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>>>> >>>>> _______________________________________________ >>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> >>>> >>>> /simple/ Pages should now be immediately invalidated when a new package is released. >>> >>> thanks Donald. Looking at the implementation, i wonder what happens if >>> after ``self._conn.commit()`` a changelog API call arrives, returns changes >>> and a client uses it to retrieve changes before the fastly-purging takes >>> place. It's still a potential race-condition or am i missing something? >>> >>> best, >>> holger >>> >>>> ----------------- >>>> Donald Stufft >>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>>> >>> >>> >> >> >> There's no way around a race condition. >> >> ``self._conn.commit()`` is what makes the changes available. If we purge prior to committing it then if someone hits the page between the purge and the self._conn.commit() then the client will see a page cached prior to the update (while the change log will appear to be updated). Essentially the same problem we have now. >> >> The current implementation does mean that if a client happens to hit between the commit and the purge they'll see old data however that's pretty unlikely. > > Purging can take a second and also depends on the network connectivity > between pypi.python.org and fastly's api to begin with. I am afraid > the race-condition is bound to happen and then hard to detect. > > Not sure how exactly pypi.python.org is deployed but could commit() use > a semaphore which also the changelog-APIs use so that the latter only > returns after purging (and them some) has happened? I don't think > mirrors would mind sometimes waiting a few seconds before the changelog* call > returns as long as the state is then consistent. > > Lastly, i think introducing a bit of internal syncing overhead to commit()/ > changelog should be ok because we have only few writes and hardly read load. Mirroring should not be affected by caching at all, as new packages mean new URLs (/pypi/name/version), so when you retrieve them there will be no cache issues. What I think you mean is this makes a race condition for pep381client, however this is a bug in pep381client, not PyPI. If you would like to submit a patch for a Paxos-based replication protocol, I'm sure Donald and I would be happy to review it. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Mon May 27 22:20:23 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 May 2013 20:20:23 +0000 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> Message-ID: <20130527202023.GJ7789@merlinux.eu> On Mon, May 27, 2013 at 12:58 -0700, Noah Kantrowitz wrote: > On May 27, 2013, at 12:18 PM, holger krekel wrote: > > > On Mon, May 27, 2013 at 14:59 -0400, Donald Stufft wrote: > >> On May 27, 2013, at 2:54 PM, holger krekel wrote: > >> > >>> On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote: > >>>> On May 27, 2013, at 12:39 PM, Donald Stufft wrote: > >>>> > >>>>> > >>>>> On May 27, 2013, at 8:08 AM, holger krekel wrote: > >>>>> > >>>>>> Hi Noah, Donald, (CC also Richard, Christian), > >>>>>> > >>>>>> i just checked with a test package and think we might have a cache > >>>>>> consistency / changelog API problem. It took me a while but here is > >>>>>> the basic thing: I uploaded a test package, changelog API reports it has > >>>>>> changed, then i go to its simple page, and some of the time the new release > >>>>>> file shows up, sometimes not. > >>>>>> > >>>>>> Tools like bandersnatch, pep381 and devpi-server (and probably others) > >>>>>> use PyPI's changelog API to determine if there are changes. It seems > >>>>>> those changes are signalled faster than they become consistently accessible > >>>>>> through the CDN. This can lead to inconsistent mirrors because when > >>>>>> the CDN has the files there is no change event anymore. Such mirrors > >>>>>> are run by companies in-house so i think it's a real problem. > >>>>>> > >>>>>> Even without mirroring there can be problems because installs are not > >>>>>> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, > >>>>>> then 2.0.0 a minute later. I had hoped that a particular ip address > >>>>>> sees things consistently. > >>>>>> > >>>>>> I am not familiar with Fastly's caching properties -- can they notify > >>>>>> about the fact that a page/file is consistently up-to-date everywhere? > >>>>>> Or can the cache be globally invalidated for a particular page/file? > >>>>>> Any other ideas? > >>>>>> > >>>>>> Failing customizing Fastly usage and also maybe for the short term, > >>>>>> is/could there be a special location provided by pypi.python.org which > >>>>>> the above tools could use to get at the actual non-cached data? We > >>>>>> could then maybe mitigate the problem through updates of the respective tools. > >>>>>> That would at least solve the problem for one of my customers i think. > >>>>>> > >>>>>> best, > >>>>>> holger > >>>>>> > >>>>>> > >>>>>> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: > >>>>>>> > >>>>>>> > >>>>>>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. > >>>>>>> > >>>>>>> --Noah > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> _______________________________________________ > >>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org > >>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Distutils-SIG maillist - Distutils-SIG at python.org > >>>>>> http://mail.python.org/mailman/listinfo/distutils-sig > >>>>> > >>>>> I mentioned it on twitter but might as well mention it here as well. > >>>>> > >>>>> Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. > >>>>> > >>>>> ----------------- > >>>>> Donald Stufft > >>>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > >>>>> > >>>>> _______________________________________________ > >>>>> Distutils-SIG maillist - Distutils-SIG at python.org > >>>>> http://mail.python.org/mailman/listinfo/distutils-sig > >>>> > >>>> > >>>> > >>>> /simple/ Pages should now be immediately invalidated when a new package is released. > >>> > >>> thanks Donald. Looking at the implementation, i wonder what happens if > >>> after ``self._conn.commit()`` a changelog API call arrives, returns changes > >>> and a client uses it to retrieve changes before the fastly-purging takes > >>> place. It's still a potential race-condition or am i missing something? > >>> > >>> best, > >>> holger > >>> > >>>> ----------------- > >>>> Donald Stufft > >>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > >>>> > >>> > >>> > >> > >> > >> There's no way around a race condition. > >> > >> ``self._conn.commit()`` is what makes the changes available. If we purge prior to committing it then if someone hits the page between the purge and the self._conn.commit() then the client will see a page cached prior to the update (while the change log will appear to be updated). Essentially the same problem we have now. > >> > >> The current implementation does mean that if a client happens to hit between the commit and the purge they'll see old data however that's pretty unlikely. > > > > Purging can take a second and also depends on the network connectivity > > between pypi.python.org and fastly's api to begin with. I am afraid > > the race-condition is bound to happen and then hard to detect. > > > > Not sure how exactly pypi.python.org is deployed but could commit() use > > a semaphore which also the changelog-APIs use so that the latter only > > returns after purging (and them some) has happened? I don't think > > mirrors would mind sometimes waiting a few seconds before the changelog* call > > returns as long as the state is then consistent. > > > > Lastly, i think introducing a bit of internal syncing overhead to commit()/ > > changelog should be ok because we have only few writes and hardly read load. > > Mirroring should not be affected by caching at all, as new packages mean new URLs (/pypi/name/version), so when you retrieve them there will be no cache issues. The simple/PROJ pages are changed, not newly created. (and yes, new release files are not so much the problem because they are new and thus retrieved from fastly on first access). > What I think you mean is this makes a race condition for pep381client, > however this is a bug in pep381client, not PyPI. If you would like to > submit a patch for a Paxos-based replication protocol, I'm sure Donald > and I would be happy to review it. I am a bit lost of what you are talking about here. The move to CDN broke things that worked before. The changelog API reported changes that could not be seen afterwards. This remains true after Donald's changes which just make it less likely but not impossible to happen. best, holger From noah at coderanger.net Mon May 27 22:34:21 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 27 May 2013 13:34:21 -0700 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <20130527202023.GJ7789@merlinux.eu> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> Message-ID: <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> On May 27, 2013, at 1:20 PM, holger krekel wrote: > On Mon, May 27, 2013 at 12:58 -0700, Noah Kantrowitz wrote: >> On May 27, 2013, at 12:18 PM, holger krekel wrote: >> >>> On Mon, May 27, 2013 at 14:59 -0400, Donald Stufft wrote: >>>> On May 27, 2013, at 2:54 PM, holger krekel wrote: >>>> >>>>> On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote: >>>>>> On May 27, 2013, at 12:39 PM, Donald Stufft wrote: >>>>>> >>>>>>> >>>>>>> On May 27, 2013, at 8:08 AM, holger krekel wrote: >>>>>>> >>>>>>>> Hi Noah, Donald, (CC also Richard, Christian), >>>>>>>> >>>>>>>> i just checked with a test package and think we might have a cache >>>>>>>> consistency / changelog API problem. It took me a while but here is >>>>>>>> the basic thing: I uploaded a test package, changelog API reports it has >>>>>>>> changed, then i go to its simple page, and some of the time the new release >>>>>>>> file shows up, sometimes not. >>>>>>>> >>>>>>>> Tools like bandersnatch, pep381 and devpi-server (and probably others) >>>>>>>> use PyPI's changelog API to determine if there are changes. It seems >>>>>>>> those changes are signalled faster than they become consistently accessible >>>>>>>> through the CDN. This can lead to inconsistent mirrors because when >>>>>>>> the CDN has the files there is no change event anymore. Such mirrors >>>>>>>> are run by companies in-house so i think it's a real problem. >>>>>>>> >>>>>>>> Even without mirroring there can be problems because installs are not >>>>>>>> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, >>>>>>>> then 2.0.0 a minute later. I had hoped that a particular ip address >>>>>>>> sees things consistently. >>>>>>>> >>>>>>>> I am not familiar with Fastly's caching properties -- can they notify >>>>>>>> about the fact that a page/file is consistently up-to-date everywhere? >>>>>>>> Or can the cache be globally invalidated for a particular page/file? >>>>>>>> Any other ideas? >>>>>>>> >>>>>>>> Failing customizing Fastly usage and also maybe for the short term, >>>>>>>> is/could there be a special location provided by pypi.python.org which >>>>>>>> the above tools could use to get at the actual non-cached data? We >>>>>>>> could then maybe mitigate the problem through updates of the respective tools. >>>>>>>> That would at least solve the problem for one of my customers i think. >>>>>>>> >>>>>>>> best, >>>>>>>> holger >>>>>>>> >>>>>>>> >>>>>>>> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. >>>>>>>>> >>>>>>>>> --Noah >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>>>> >>>>>>> I mentioned it on twitter but might as well mention it here as well. >>>>>>> >>>>>>> Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. >>>>>>> >>>>>>> ----------------- >>>>>>> Donald Stufft >>>>>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>>> >>>>>> >>>>>> >>>>>> /simple/ Pages should now be immediately invalidated when a new package is released. >>>>> >>>>> thanks Donald. Looking at the implementation, i wonder what happens if >>>>> after ``self._conn.commit()`` a changelog API call arrives, returns changes >>>>> and a client uses it to retrieve changes before the fastly-purging takes >>>>> place. It's still a potential race-condition or am i missing something? >>>>> >>>>> best, >>>>> holger >>>>> >>>>>> ----------------- >>>>>> Donald Stufft >>>>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>>>>> >>>>> >>>>> >>>> >>>> >>>> There's no way around a race condition. >>>> >>>> ``self._conn.commit()`` is what makes the changes available. If we purge prior to committing it then if someone hits the page between the purge and the self._conn.commit() then the client will see a page cached prior to the update (while the change log will appear to be updated). Essentially the same problem we have now. >>>> >>>> The current implementation does mean that if a client happens to hit between the commit and the purge they'll see old data however that's pretty unlikely. >>> >>> Purging can take a second and also depends on the network connectivity >>> between pypi.python.org and fastly's api to begin with. I am afraid >>> the race-condition is bound to happen and then hard to detect. >>> >>> Not sure how exactly pypi.python.org is deployed but could commit() use >>> a semaphore which also the changelog-APIs use so that the latter only >>> returns after purging (and them some) has happened? I don't think >>> mirrors would mind sometimes waiting a few seconds before the changelog* call >>> returns as long as the state is then consistent. >>> >>> Lastly, i think introducing a bit of internal syncing overhead to commit()/ >>> changelog should be ok because we have only few writes and hardly read load. >> >> Mirroring should not be affected by caching at all, as new packages mean new URLs (/pypi/name/version), so when you retrieve them there will be no cache issues. > > The simple/PROJ pages are changed, not newly created. (and yes, > new release files are not so much the problem because they are new > and thus retrieved from fastly on first access). Yes, pep381client is fundamentally incompatible with the future of PyPI's infrastructure. Sorry, this will not be changed at this point. If people would like to continue to operate mirrors, they will need to transition to use the API to access package information, fetch updated files, and rebuild any relevant index data. For example, this is how Donald's crate.io mirror operates. Using the current strategy of scraping the simple/ pages will continue to work, you just need to retry failed requests until they succeed (and check that the per-project pages match the version you expect from the change log, consider it a failure if they do not). This is just a stopgap though, and should not be considered a long-term solution. > >> What I think you mean is this makes a race condition for pep381client, >> however this is a bug in pep381client, not PyPI. If you would like to >> submit a patch for a Paxos-based replication protocol, I'm sure Donald >> and I would be happy to review it. > > I am a bit lost of what you are talking about here. > > The move to CDN broke things that worked before. The changelog API > reported changes that could not be seen afterwards. This remains true > after Donald's changes which just make it less likely but not impossible > to happen. They worked effectively by accident, not because it was correct. Had I understood how backwards the pep381 systems are, I would have alerted you all sooner, I apologize for this lapse. I am happy to talk about how to correctly use the PyPI API with anyone that has questions, or discuss more advanced replication options that will be free of race conditions in a distributed (yes, PyPI is a distributed database now) environment. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 27 22:41:17 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 May 2013 16:41:17 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> Message-ID: On May 27, 2013, at 4:34 PM, Noah Kantrowitz wrote: > > On May 27, 2013, at 1:20 PM, holger krekel wrote: > >> On Mon, May 27, 2013 at 12:58 -0700, Noah Kantrowitz wrote: >>> On May 27, 2013, at 12:18 PM, holger krekel wrote: >>> >>>> On Mon, May 27, 2013 at 14:59 -0400, Donald Stufft wrote: >>>>> On May 27, 2013, at 2:54 PM, holger krekel wrote: >>>>> >>>>>> On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote: >>>>>>> On May 27, 2013, at 12:39 PM, Donald Stufft wrote: >>>>>>> >>>>>>>> >>>>>>>> On May 27, 2013, at 8:08 AM, holger krekel wrote: >>>>>>>> >>>>>>>>> Hi Noah, Donald, (CC also Richard, Christian), >>>>>>>>> >>>>>>>>> i just checked with a test package and think we might have a cache >>>>>>>>> consistency / changelog API problem. It took me a while but here is >>>>>>>>> the basic thing: I uploaded a test package, changelog API reports it has >>>>>>>>> changed, then i go to its simple page, and some of the time the new release >>>>>>>>> file shows up, sometimes not. >>>>>>>>> >>>>>>>>> Tools like bandersnatch, pep381 and devpi-server (and probably others) >>>>>>>>> use PyPI's changelog API to determine if there are changes. It seems >>>>>>>>> those changes are signalled faster than they become consistently accessible >>>>>>>>> through the CDN. This can lead to inconsistent mirrors because when >>>>>>>>> the CDN has the files there is no change event anymore. Such mirrors >>>>>>>>> are run by companies in-house so i think it's a real problem. >>>>>>>>> >>>>>>>>> Even without mirroring there can be problems because installs are not >>>>>>>>> directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1, >>>>>>>>> then 2.0.0 a minute later. I had hoped that a particular ip address >>>>>>>>> sees things consistently. >>>>>>>>> >>>>>>>>> I am not familiar with Fastly's caching properties -- can they notify >>>>>>>>> about the fact that a page/file is consistently up-to-date everywhere? >>>>>>>>> Or can the cache be globally invalidated for a particular page/file? >>>>>>>>> Any other ideas? >>>>>>>>> >>>>>>>>> Failing customizing Fastly usage and also maybe for the short term, >>>>>>>>> is/could there be a special location provided by pypi.python.org which >>>>>>>>> the above tools could use to get at the actual non-cached data? We >>>>>>>>> could then maybe mitigate the problem through updates of the respective tools. >>>>>>>>> That would at least solve the problem for one of my customers i think. >>>>>>>>> >>>>>>>>> best, >>>>>>>>> holger >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. >>>>>>>>>> >>>>>>>>>> --Noah >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>>>>> >>>>>>>> I mentioned it on twitter but might as well mention it here as well. >>>>>>>> >>>>>>>> Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI. >>>>>>>> >>>>>>>> ----------------- >>>>>>>> Donald Stufft >>>>>>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>>>> http://mail.python.org/mailman/listinfo/distutils-sig >>>>>>> >>>>>>> >>>>>>> >>>>>>> /simple/ Pages should now be immediately invalidated when a new package is released. >>>>>> >>>>>> thanks Donald. Looking at the implementation, i wonder what happens if >>>>>> after ``self._conn.commit()`` a changelog API call arrives, returns changes >>>>>> and a client uses it to retrieve changes before the fastly-purging takes >>>>>> place. It's still a potential race-condition or am i missing something? >>>>>> >>>>>> best, >>>>>> holger >>>>>> >>>>>>> ----------------- >>>>>>> Donald Stufft >>>>>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> There's no way around a race condition. >>>>> >>>>> ``self._conn.commit()`` is what makes the changes available. If we purge prior to committing it then if someone hits the page between the purge and the self._conn.commit() then the client will see a page cached prior to the update (while the change log will appear to be updated). Essentially the same problem we have now. >>>>> >>>>> The current implementation does mean that if a client happens to hit between the commit and the purge they'll see old data however that's pretty unlikely. >>>> >>>> Purging can take a second and also depends on the network connectivity >>>> between pypi.python.org and fastly's api to begin with. I am afraid >>>> the race-condition is bound to happen and then hard to detect. >>>> >>>> Not sure how exactly pypi.python.org is deployed but could commit() use >>>> a semaphore which also the changelog-APIs use so that the latter only >>>> returns after purging (and them some) has happened? I don't think >>>> mirrors would mind sometimes waiting a few seconds before the changelog* call >>>> returns as long as the state is then consistent. >>>> >>>> Lastly, i think introducing a bit of internal syncing overhead to commit()/ >>>> changelog should be ok because we have only few writes and hardly read load. >>> >>> Mirroring should not be affected by caching at all, as new packages mean new URLs (/pypi/name/version), so when you retrieve them there will be no cache issues. >> >> The simple/PROJ pages are changed, not newly created. (and yes, >> new release files are not so much the problem because they are new >> and thus retrieved from fastly on first access). > > Yes, pep381client is fundamentally incompatible with the future of PyPI's infrastructure. Sorry, this will not be changed at this point. If people would like to continue to operate mirrors, they will need to transition to use the API to access package information, fetch updated files, and rebuild any relevant index data. For example, this is how Donald's crate.io mirror operates. Using the current strategy of scraping the simple/ pages will continue to work, you just need to retry failed requests until they succeed (and check that the per-project pages match the version you expect from the change log, consider it a failure if they do not). This is just a stopgap though, and should not be considered a long-term solution. > >> >>> What I think you mean is this makes a race condition for pep381client, >>> however this is a bug in pep381client, not PyPI. If you would like to >>> submit a patch for a Paxos-based replication protocol, I'm sure Donald >>> and I would be happy to review it. >> >> I am a bit lost of what you are talking about here. >> >> The move to CDN broke things that worked before. The changelog API >> reported changes that could not be seen afterwards. This remains true >> after Donald's changes which just make it less likely but not impossible >> to happen. > > They worked effectively by accident, not because it was correct. Had I understood how backwards the pep381 systems are, I would have alerted you all sooner, I apologize for this lapse. I am happy to talk about how to correctly use the PyPI API with anyone that has questions, or discuss more advanced replication options that will be free of race conditions in a distributed (yes, PyPI is a distributed database now) environment. > > --Noah > Just to assure folks. I do consider Mirroring a first class citizen and an important feature. Better support will be coming for it. However this relatively minor issue requiring that clients get a little smarter to use the current system is a regrettable requirement to make installation from PyPI not suck for a large swathe of people. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ralf at systemexit.de Mon May 27 23:21:15 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Mon, 27 May 2013 23:21:15 +0200 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> (Noah Kantrowitz's message of "Sun, 26 May 2013 10:34:58 -0700") References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> Message-ID: <877giki0fo.fsf@brainbot.com> Noah Kantrowitz writes: > > > but seriously, at long last today it was my honor to throw the DNS > switch to move PyPI to the Fastly caching CDN. I would like to thank > Donald Stufft for doing much of the heavy lifting on the PyPI side, > and to Fastly for graciously offering to host us. What does this mean > for everyone? Well the biggest change is PyPI should get a whole lot > faster. There are two major downsides however. There will now be a > delay of several minutes in some cases between updating a package and > having it be installable, and download counts will now be even more > incorrect than they were before. The PyPI admins are discussing what > to do about download counts long-term, but for now we all feel that > the performance and availability benefits outweigh the loss. If anyone > has any questions, or hears anything about issues with PyPI please > don't hesitate to contact me. > > --Noah the xmlrpc api is broken when using http 1.0. the second call to curl uses http/1.0 and returns an empty response: $ cat >body.txt < package_releases e EOF $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" 1.4.5 $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" $ From noah at coderanger.net Mon May 27 23:24:53 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 27 May 2013 14:24:53 -0700 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <877giki0fo.fsf@brainbot.com> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> Message-ID: <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> On May 27, 2013, at 2:21 PM, Ralf Schmitt wrote: > Noah Kantrowitz writes: > >> >> >> but seriously, at long last today it was my honor to throw the DNS >> switch to move PyPI to the Fastly caching CDN. I would like to thank >> Donald Stufft for doing much of the heavy lifting on the PyPI side, >> and to Fastly for graciously offering to host us. What does this mean >> for everyone? Well the biggest change is PyPI should get a whole lot >> faster. There are two major downsides however. There will now be a >> delay of several minutes in some cases between updating a package and >> having it be installable, and download counts will now be even more >> incorrect than they were before. The PyPI admins are discussing what >> to do about download counts long-term, but for now we all feel that >> the performance and availability benefits outweigh the loss. If anyone >> has any questions, or hears anything about issues with PyPI please >> don't hesitate to contact me. >> >> --Noah > > > the xmlrpc api is broken when using http 1.0. the second call to curl > uses http/1.0 and returns an empty response: > > $ cat >body.txt < > > package_releases > > > e > > > > EOF > $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" > > > > > > 1.4.5 > > > > > $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" > $ We have not supported HTTP 1.0 for quite some time. Even before the CDN move, we used the Host header to route between different HAProxy server blocks on the load balancers. I'm unaware of any reason people would be using HTTP 1.0 clients at this point, HTTP 1.1 has been a standard for 14 years now. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 27 23:28:03 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 May 2013 17:28:03 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <06FE1104-014B-4EA1-9CAF-6BEC0FAFD4F1@mac.com> References: <06FE1104-014B-4EA1-9CAF-6BEC0FAFD4F1@mac.com> Message-ID: <14C2D779-8D99-4036-948A-7BF367A1DCB6@stufft.io> On May 27, 2013, at 10:44 AM, Ronald Oussoren wrote: > > On 27 May, 2013, at 13:36, Nick Coghlan wrote: > >> After preliminary reviews by Donald and Daniel, I have now pushed the >> first complete draft of the JSON-based metadata 2.0 proposal to >> python.org >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > Could platform_release be added to the set of environment markers? On OSX platform.release() returns a kernel version number, while platform.version() returns a string that cannot easily be used as a marker. It might also be nice to have markers for the marketing name and version of the OS next to the kernel name and release (for example Solaris 8 vs. SunOS 5.8, or darwin 12.3.0 vs. OSX 10.8.3), but that would probably need a PEP of its own to add the functionality to the stdlib before it is used in metadata. > > BTW. Why can marker expressions use only a limited subset of operators? In particular <, <=, > and >= are not allowed and would be nice to have to specify packages that need a minimal platform version (parts of PyObjC have a dependency on the OSX release because the wrapped library is available in a subset of releases). Listing all OS releases would be fairly useless, platform.release() is too specific to use with '==' as new patch releases of OSX can introduce new micro versions of the kernel. > >> PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ > > The versioning spec mentions that distribution tools may refuse to publish distributions that pin the versions of dependencies. I understand why this is needed, and agree in general, but have a usecase that I don't know how to express without pinning. > > In particular, PyObjC consists of a number of distributions (pyobjc-core, pyobjc-framework-Cocoa, ...) and an umbrella package (pyobjc) what depends on the various distributions to make it easier to install all of PyObjC. The umbrella package currently pins the versions of subpackages to ensure that "pip install pyobjc==2.5.1" installs exactly that version of the entire project. When I'd use the "compatible release" specifier I can no longer easily ensure that users can install an exact version of the entire project, other than by hacking the system: specify a compatible version with an additional level that isn't used by the project (for example ~=2.5.2.0). What is the correct way to create an umberella project without getting yelled at by distribution tools? It's unlikely PyPI will get more than a warning for ``==``, `is` comparisons might be disallowed? Not sure. > > Ronald > >> >> With the rationale and commentary, they're over 3000 lines between >> them, so I'm not attaching them here. >> >> The rationale for many of the changes is at the end of each PEP, along >> with some comments on features that I have either rejected or >> deliberately chosen to defer to the next revision of the metadata (at >> the earliest). >> >> Those with BitBucket accounts may also comment inline on the drafts here: >> >> PEP 426: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0426.txt?at=default >> PEP 440: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0440.txt?at=default >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ralf at systemexit.de Tue May 28 00:02:38 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Tue, 28 May 2013 00:02:38 +0200 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> (Noah Kantrowitz's message of "Mon, 27 May 2013 14:24:53 -0700") References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> Message-ID: <87obbwgjy9.fsf@brainbot.com> Noah Kantrowitz writes: >> >> >> the xmlrpc api is broken when using http 1.0. the second call to curl >> uses http/1.0 and returns an empty response: >> >> $ cat >body.txt <> >> >> package_releases >> >> >> e >> >> >> >> EOF >> $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" >> >> >> >> >> >> 1.4.5 >> >> >> >> >> $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" >> $ > > We have not supported HTTP 1.0 for quite some time. well, it has been working until at least last friday. > Even before the CDN move, we used the Host header to route between > different HAProxy server blocks on the load balancers. I'm unaware of > any reason people would be using HTTP 1.0 clients at this point, HTTP > 1.1 has been a standard for 14 years now. the fact that you are unaware of any reason people would be using HTTP 1.0 is not a good reason not to support it. python 2.6 xmlrpclib uses HTTP/1.0! -- Cheers Ralf From donald at stufft.io Tue May 28 03:33:50 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 May 2013 21:33:50 -0400 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <87obbwgjy9.fsf@brainbot.com> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> Message-ID: <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> On May 27, 2013, at 6:02 PM, Ralf Schmitt wrote: > Noah Kantrowitz writes: > >>> >>> >>> the xmlrpc api is broken when using http 1.0. the second call to curl >>> uses http/1.0 and returns an empty response: >>> >>> $ cat >body.txt <>> >>> >>> package_releases >>> >>> >>> e >>> >>> >>> >>> EOF >>> $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" >>> >>> >>> >>> >>> >>> 1.4.5 >>> >>> >>> >>> >>> $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" >>> $ >> >> We have not supported HTTP 1.0 for quite some time. > > well, it has been working until at least last friday. > >> Even before the CDN move, we used the Host header to route between >> different HAProxy server blocks on the load balancers. I'm unaware of >> any reason people would be using HTTP 1.0 clients at this point, HTTP >> 1.1 has been a standard for 14 years now. > > the fact that you are unaware of any reason people would be using HTTP > 1.0 is not a good reason not to support it. > > python 2.6 xmlrpclib uses HTTP/1.0! Maybe I misunderstand, but xmlrpclib on Python2.6 and the new CDN work fine together. $ python2.6 (pypi) Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00) [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import xmlrpclib >>> s = xmlrpclib.ServerProxy("https://pypi.python.org/pypi") >>> s >>> s.user_packages("dstufft") [...] >>> # Just incase we aren't getting the CDN on pypi.python.org >>> s = xmlrpclib.ServerProxy("https://pypi.a.ssl.fastly.net/pypi") >>> s.user_packages("dstufft") [...] > > -- > Cheers > Ralf > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue May 28 03:53:29 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 May 2013 21:53:29 -0400 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> Message-ID: <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> On May 27, 2013, at 9:33 PM, Donald Stufft wrote: > > On May 27, 2013, at 6:02 PM, Ralf Schmitt wrote: > >> Noah Kantrowitz writes: >> >>>> >>>> >>>> the xmlrpc api is broken when using http 1.0. the second call to curl >>>> uses http/1.0 and returns an empty response: >>>> >>>> $ cat >body.txt <>>> >>>> >>>> package_releases >>>> >>>> >>>> e >>>> >>>> >>>> >>>> EOF >>>> $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" >>>> >>>> >>>> >>>> >>>> >>>> 1.4.5 >>>> >>>> >>>> >>>> >>>> $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header "Content-Type:text/xml" >>>> $ >>> >>> We have not supported HTTP 1.0 for quite some time. >> >> well, it has been working until at least last friday. >> >>> Even before the CDN move, we used the Host header to route between >>> different HAProxy server blocks on the load balancers. I'm unaware of >>> any reason people would be using HTTP 1.0 clients at this point, HTTP >>> 1.1 has been a standard for 14 years now. >> >> the fact that you are unaware of any reason people would be using HTTP >> 1.0 is not a good reason not to support it. >> >> python 2.6 xmlrpclib uses HTTP/1.0! > > Maybe I misunderstand, but xmlrpclib on Python2.6 and the new CDN work fine together. > > $ python2.6 (pypi) > Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00) > [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import xmlrpclib > >>> s = xmlrpclib.ServerProxy("https://pypi.python.org/pypi") > >>> s > > >>> s.user_packages("dstufft") > [...] > >>> # Just incase we aren't getting the CDN on pypi.python.org > >>> s = xmlrpclib.ServerProxy("https://pypi.a.ssl.fastly.net/pypi") > >>> s.user_packages("dstufft") > [...] > >> >> -- >> Cheers >> Ralf >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Figured it out. Use HTTPS. $ curl -0 -X POST -d @body.txt https://pypi.python.org/pypi --header "Content-Type:text/xml" 1.4.5 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue May 28 08:42:05 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 May 2013 07:42:05 +0100 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> Message-ID: On 28 May 2013 02:53, Donald Stufft wrote: > Figured it out. > > Use HTTPS. > Can I suggest that if the new CDN means that there are additional restrictions on what is supported (I've used the XMLRPC API without https in one-off scripts in the past) then the officially supported API should be properly documented once and for all in a PEP, including some sort of "what's new" or "rationale" section describing the various changes that have occurred recently and their impact on user code? I'm purely a casual user of the PyPI API and the discussion of these changes haa mostly gone over my head. The one thing I've taken away from it is that I may get problems if I just google for sample code to use. For example, the above comment implies that http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal documentation that the XMLRPC API has) is wrong (as it uses http). I do appreciate all the work that is going on to improve the PyPI infrastructure. I'm not saying the changes should be reverted, just that the consequences should be clearly explained. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue May 28 08:57:13 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 May 2013 06:57:13 +0000 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> Message-ID: <20130528065713.GL7789@merlinux.eu> On Tue, May 28, 2013 at 07:42 +0100, Paul Moore wrote: > On 28 May 2013 02:53, Donald Stufft wrote: > > > Figured it out. > > > > Use HTTPS. > > > > Can I suggest that if the new CDN means that there are additional > restrictions on what is supported (I've used the XMLRPC API without https > in one-off scripts in the past) then the officially supported API should be > properly documented once and for all in a PEP, including some sort of > "what's new" or "rationale" section describing the various changes that > have occurred recently and their impact on user code? I second this. I am building tools that interact with PyPI and people and customers are using them. I don't want to find a switch announced which breaks them and then hear "sorry, that's the future now" without this future being documented and discussed before the fact. The PyPI infrastructure and its supported tool interactions today are as important as evolving the language itself so PEPs are warranted. As with PEP438 i am willing to help this process. > I'm purely a casual user of the PyPI API and the discussion of these > changes haa mostly gone over my head. The one thing I've taken away from it > is that I may get problems if I just google for sample code to use. For > example, the above comment implies that > http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal > documentation that the XMLRPC API has) is wrong (as it uses http). > > I do appreciate all the work that is going on to improve the PyPI > infrastructure. I'm not saying the changes should be reverted, just that > the consequences should be clearly explained. I also appreciate Noah's and Donald's CDN work here, up to the point where it breaks things for unclear reasons. Reasons which might very well be valid, nevertheless! best, holger From holger at merlinux.eu Tue May 28 11:25:43 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 May 2013 09:25:43 +0000 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> References: <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Message-ID: <20130528092543.GT7789@merlinux.eu> On Tue, May 28, 2013 at 11:04 +0200, Christian Theune wrote: > On 27. May2013, at 10:41 PM, Donald Stufft wrote: > > Just to assure folks. I do consider Mirroring a first class citizen and an important feature. > > Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon. > > I agree with Holgers points. > > I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. > > My current client implementation is pretty simple and has had reliable semantics until now. > > A couple of things I noticed in the discussion that I'd like to point out: > > - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken. > > - Release files are replaced all the time. > > The semantics that I like to keep with the mirrors is this: > > When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X (but maybe also partial changes from X+n). > > I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. > > I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting. > > So - what's the next step that can happen ASAP? The immediate way to get around the CDN/mirroring problems and to revert to the pre-CDN consistency level, is to use the same access that fastly uses to get updates from pypi.python.org, namely a request on front.python.org with a host-header. I have this info from Donald with the cave-eat that it's not guaranteed to remain possible. Maybe Noah could agree to not remove this facility without the current actors being on board for changes? (i am also fine to have a dedicated domain instead of course). Once this is settled, we can move on to fix current tools and deployments and afterwards think about future improvements without the current urgency. holger > Christian > > -- > Christian Theune ? ct at gocept.com > gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany > http://gocept.com ? Tel +49 345 1229889-7 > Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations > From donald at stufft.io Tue May 28 13:48:06 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 07:48:06 -0400 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> Message-ID: On May 28, 2013, at 2:42 AM, Paul Moore wrote: > On 28 May 2013 02:53, Donald Stufft wrote: > Figured it out. > > Use HTTPS. > > Can I suggest that if the new CDN means that there are additional restrictions on what is supported (I've used the XMLRPC API without https in one-off scripts in the past) then the officially supported API should be properly documented once and for all in a PEP, including some sort of "what's new" or "rationale" section describing the various changes that have occurred recently and their impact on user code? > > I'm purely a casual user of the PyPI API and the discussion of these changes haa mostly gone over my head. The one thing I've taken away from it is that I may get problems if I just google for sample code to use. For example, the above comment implies that http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal documentation that the XMLRPC API has) is wrong (as it uses http). > > I do appreciate all the work that is going on to improve the PyPI infrastructure. I'm not saying the changes should be reverted, just that the consequences should be clearly explained. > > Paul. To be quite honest the HTTP 1.0 + HTTP issue simply wasn't discovered in testing. The http url works fine on Python 2.7 (which I'm assuming uses HTTP 1.1). I'm not completely happy that HTTP is broken in Python2.6 (and I'm assuming earlier) and have it on my list to see if there's anything that can be done. THat being said the most future compatible way will be to use the HTTPS url for any interaction (and ideally verify the SSL, but the built in XMLRPC library doesn't do that). My "Use HTTPS" was more to speak how to solve the issue *right now*. Documentation should be updated to point to HTTPS though. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ct at gocept.com Tue May 28 11:04:25 2013 From: ct at gocept.com (Christian Theune) Date: Tue, 28 May 2013 11:04:25 +0200 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> Message-ID: <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Hi, On 27. May2013, at 10:41 PM, Donald Stufft wrote: > Just to assure folks. I do consider Mirroring a first class citizen and an important feature. Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon. I agree with Holgers points. I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. My current client implementation is pretty simple and has had reliable semantics until now. A couple of things I noticed in the discussion that I'd like to point out: - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken. - Release files are replaced all the time. The semantics that I like to keep with the mirrors is this: When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X (but maybe also partial changes from X+n). I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting. So - what's the next step that can happen ASAP? Christian -- Christian Theune ? ct at gocept.com gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany http://gocept.com ? Tel +49 345 1229889-7 Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 535 bytes Desc: Message signed with OpenPGP using GPGMail URL: From domen at dev.si Mon May 27 18:04:39 2013 From: domen at dev.si (=?UTF-8?B?RG9tZW4gS2/FvmFy?=) Date: Mon, 27 May 2013 18:04:39 +0200 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130527154636.GF7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> <20130527075934.GZ7789@merlinux.eu> <87y5b0u8ej.fsf@eve.Speedport_W_921V_1_21_000> <20130527154636.GF7789@merlinux.eu> Message-ID: I'll also be at EP and can help to explain how Nix could solve the isolation problem. On Mon, May 27, 2013 at 5:46 PM, holger krekel wrote: > Hi Florian, > > On Mon, May 27, 2013 at 10:36 +0200, Florian Friesdorf wrote: > > Hi Holger, > > > > holger krekel writes: > > > On Mon, May 27, 2013 at 17:41 +1000, Nick Coghlan wrote: > > >> On Mon, May 27, 2013 at 5:27 PM, holger krekel > wrote: > > >> > Not having download counts maybe lets us think harder about > > >> > better metrics. The number of projects using a package as a dep > > >> > might be one. > > >> > > >> With the current downside being that it's hard for PyPI to figure out > > >> that number, too :) > > > > > > Yip. But something like Vinaj's red-dove approach or Marius' > get_deps.py > > > could provide a base. We might think about a docker instance which > > > could allow to quickly spawn new light VMs so we can isolate setup.py > runs. > > > (Yes, it's only Linux but it'd be a start). > > > > nix and nixpkgs allow this isolation on-top off linux, freebsd, OS X and > > theoretically also cygwin (not sure how good cygwin is supported at the > > moment). > > > > http://nixos.org/nix/ > > http://nixos.org/nixpkgs/ > > > > From nixos.org: > > Nix is a purely functional package manager. This means that it can > > ensure that an upgrade to one package cannot break others, that you can > > always roll back to previous version, that multiple versions of a > > package can coexist on the same system, and much more. > > > > Nixpkgs is a large collection of packages that can be installed with the > > Nix package manager. > > Interesting stuff, didn't know about it. > > Did you post this as a suggestion for provisioning an environment to > run setup.py (on nix-supported platforms)? If so, i am not sure how it > would help exactly. I guess myself i'd aim for a 80% solution for > discovering > dependencies first. Simplest/quickest wins there :) > > > >> Agreed it would be a good number to publish once it's more readily > > >> available, too. > > > > > > I think "dep" numbers are mostly interesting for libraries, not so > > > much for applications like django or pyramid or tools like nose/pytest. > > > > > > Another more practical data point would be "does this package even > > > install on win32/linux/osx py26/py27/py33" and even better, do its > automated > > > tests pass? > > > > http://hydra.nixos.org/build/5062796 > > > > > If we could evolve to have this info published on pypi.python.org > > > it would be quite useful i think. I am actually currently implementing > > > a system which enables this (the "devpi" system) so i don't mean this > all just > > > as "nice to have" theory. I aim to present the status of this work > > > at EuroPython. > > > > Nice! Looking forward to that. > > > > If you have any questions about nix/nixpkgs/nixos, especially about the > > way python packages are packaged, please let me know. Also, it's not set > > in stone. > > are you going to be at EP? It's a long conference and i am more than > happy to sit together on this topic for a bit sometimes. > > best, > holger > > > > Personally, I'd love to see hydra.python.org providing builds of all > > pypi packages and would be happy to help. Also including Domen and Rok > > for whome I assume the same. > > > > > > You might have other tools that are better suited for you. > > > > regards > > florian > > -- > > Florian Friesdorf > > GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083 > > Jabber/XMPP: flo at chaoflow.net > > IRC: chaoflow on freenode,ircnet,blafasel,OFTC > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue May 28 14:07:49 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 08:07:49 -0400 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: <20130528065713.GL7789@merlinux.eu> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> <20130528065713.GL7789@merlinux.eu> Message-ID: On May 28, 2013, at 2:57 AM, holger krekel wrote: > On Tue, May 28, 2013 at 07:42 +0100, Paul Moore wrote: >> On 28 May 2013 02:53, Donald Stufft wrote: >> >>> Figured it out. >>> >>> Use HTTPS. >>> >> >> Can I suggest that if the new CDN means that there are additional >> restrictions on what is supported (I've used the XMLRPC API without https >> in one-off scripts in the past) then the officially supported API should be >> properly documented once and for all in a PEP, including some sort of >> "what's new" or "rationale" section describing the various changes that >> have occurred recently and their impact on user code? > > I second this. I am building tools that interact with PyPI and people > and customers are using them. I don't want to find a switch announced > which breaks them and then hear "sorry, that's the future now" without > this future being documented and discussed before the fact. The PyPI > infrastructure and its supported tool interactions today are as important as > evolving the language itself so PEPs are warranted. As with PEP438 i am > willing to help this process. > >> I'm purely a casual user of the PyPI API and the discussion of these >> changes haa mostly gone over my head. The one thing I've taken away from it >> is that I may get problems if I just google for sample code to use. For >> example, the above comment implies that >> http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal >> documentation that the XMLRPC API has) is wrong (as it uses http). >> >> I do appreciate all the work that is going on to improve the PyPI >> infrastructure. I'm not saying the changes should be reverted, just that >> the consequences should be clearly explained. > > I also appreciate Noah's and Donald's CDN work here, up to the point where > it breaks things for unclear reasons. Reasons which might very well > be valid, nevertheless! > > best, > holger Moving to a CDN has been discussed before on either catalog-sig or distutils-sig (Can't recall which offhand). Weekly status updates were posted to the infrastructure list as well as the communication between us and Fastly as we ironed out SSL issues. The mirroring issue pre-invalidation was quickly corrected. We now invalidate and we are looking at a window that is at most a few seconds large. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue May 28 14:20:25 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 08:20:25 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Message-ID: On May 28, 2013, at 5:04 AM, Christian Theune wrote: > Hi, > > > On 27. May2013, at 10:41 PM, Donald Stufft wrote: >> Just to assure folks. I do consider Mirroring a first class citizen and an important feature. > > Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon. > > I agree with Holgers points. > > I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. > > My current client implementation is pretty simple and has had reliable semantics until now. > > A couple of things I noticed in the discussion that I'd like to point out: > > - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken. > > - Release files are replaced all the time. > > The semantics that I like to keep with the mirrors is this: > > When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X (but maybe also partial changes from X+n). > > I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. > > I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting. > > So - what's the next step that can happen ASAP? Options) 1) When mirroring retain N minutes worth of old serials and redo them. Mirroring is idempotent you can repeat it with no negative side effects. Conditional HTTP requests should also be supported to minimize the bandwidth. 2) Wait a few seconds after fetching the change log to begin processing. 3) Use front.python.org with the pypi.python.org HOST header with the caveat this is not guaranteed to be stable in the long term. 4) ??? Of them 1) is more likely to give you the best results within the constraints of HTTP. All it takes is someone to run your mirroring script behind a caching proxy and pre-CDN you'd have the exact situation we have now. Mirroring is in a bad state because it comes (and has always) with absolutely no guarantees of consistency. You dismiss the issues of having serial n+1 changes, but that is a serious problem. If you fetch up to serial N of package1 which has the released version of 1.0, and then you fetch serial N+2 of package2 which has a hard requirement on package 1.1 (which was released in serial N+1) you now have packages that are not installable via your mirror because of inconsistent state. If someone comes up with a better option that doesn't require a large rearch of the storage code in PyPI I'm happy to review and deploy it. > > Christian > > -- > Christian Theune ? ct at gocept.com > gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany > http://gocept.com ? Tel +49 345 1229889-7 > Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Tue May 28 14:26:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 May 2013 22:26:04 +1000 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> <20130528065713.GL7789@merlinux.eu> Message-ID: On Tue, May 28, 2013 at 10:07 PM, Donald Stufft wrote: > Moving to a CDN has been discussed before on either catalog-sig or > distutils-sig (Can't recall which offhand). > > Weekly status updates were posted to the infrastructure list as well as the > communication between us and Fastly as we ironed out SSL issues. > > The mirroring issue pre-invalidation was quickly corrected. We now > invalidate and we are looking at a window that is at most a few seconds > large. One of the things I (successfully) advocated for at PyCon US was to open up the PEP process to cover things where python-dev aren't directly involved, but we need an official avenue for publication of significant changes in the Python ecosystem (with my main aim being to empower distutils-sig as a place where we could actually making final decisions about the evolution of the packaging ecosystem). Given that, an Informational PEP with Discussion-To set to infrastructure-sig and Noah as BDFL-Delegate would be an eminently suitable way of keeping PyPI users and mirror operators that *aren't* following infrastructure-sig informed of upcoming changes that may impact the operation of PyPI clients. infrastructure-sig has historically just been for backend hosting details, without significant impact to *client* facing behaviour - while I think it's fine to change that, it's also understandable that most developers of PyPI clients wouldn't be aware of upcoming changes that have only been discussed in detail on that list. So, as Holger said, great work and thanks for your efforts, but good communication does matter with these things. People don't like surprises, even well intentioned ones :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Tue May 28 14:47:19 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 08:47:19 -0400 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> <20130528065713.GL7789@merlinux.eu> Message-ID: <856A3B80-E366-487C-8F5A-5E0EC1D849EA@stufft.io> On May 28, 2013, at 8:26 AM, Nick Coghlan wrote: > On Tue, May 28, 2013 at 10:07 PM, Donald Stufft wrote: >> Moving to a CDN has been discussed before on either catalog-sig or >> distutils-sig (Can't recall which offhand). >> >> Weekly status updates were posted to the infrastructure list as well as the >> communication between us and Fastly as we ironed out SSL issues. >> >> The mirroring issue pre-invalidation was quickly corrected. We now >> invalidate and we are looking at a window that is at most a few seconds >> large. > > One of the things I (successfully) advocated for at PyCon US was to > open up the PEP process to cover things where python-dev aren't > directly involved, but we need an official avenue for publication of > significant changes in the Python ecosystem (with my main aim being to > empower distutils-sig as a place where we could actually making final > decisions about the evolution of the packaging ecosystem). > > Given that, an Informational PEP with Discussion-To set to > infrastructure-sig and Noah as BDFL-Delegate would be an eminently > suitable way of keeping PyPI users and mirror operators that *aren't* > following infrastructure-sig informed of upcoming changes that may > impact the operation of PyPI clients. > > infrastructure-sig has historically just been for backend hosting > details, without significant impact to *client* facing behaviour - > while I think it's fine to change that, it's also understandable that > most developers of PyPI clients wouldn't be aware of upcoming changes > that have only been discussed in detail on that list. > It is only a significant change if you make invalid assumptions about HTTP and consistent state between two requests. If you want to rely on that then let's talk about a system where we can reliably promise that. > So, as Holger said, great work and thanks for your efforts, but good > communication does matter with these things. People don't like > surprises, even well intentioned ones :) > Point taken. In the future ill post any infrastructure upgrades I'm involved in not only to the infrastructure list but also to distutils sig. > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Tue May 28 14:48:25 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 May 2013 14:48:25 +0200 Subject: [Distutils] [Infrastructure] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> <20130528065713.GL7789@merlinux.eu> Message-ID: <51A4A799.6050603@egenix.com> On 28.05.2013 14:26, Nick Coghlan wrote: > On Tue, May 28, 2013 at 10:07 PM, Donald Stufft wrote: >> Moving to a CDN has been discussed before on either catalog-sig or >> distutils-sig (Can't recall which offhand). >> >> Weekly status updates were posted to the infrastructure list as well as the >> communication between us and Fastly as we ironed out SSL issues. >> >> The mirroring issue pre-invalidation was quickly corrected. We now >> invalidate and we are looking at a window that is at most a few seconds >> large. > > One of the things I (successfully) advocated for at PyCon US was to > open up the PEP process to cover things where python-dev aren't > directly involved, but we need an official avenue for publication of > significant changes in the Python ecosystem (with my main aim being to > empower distutils-sig as a place where we could actually making final > decisions about the evolution of the packaging ecosystem). > > Given that, an Informational PEP with Discussion-To set to > infrastructure-sig and Noah as BDFL-Delegate would be an eminently > suitable way of keeping PyPI users and mirror operators that *aren't* > following infrastructure-sig informed of upcoming changes that may > impact the operation of PyPI clients. > > infrastructure-sig has historically just been for backend hosting > details, without significant impact to *client* facing behaviour - > while I think it's fine to change that, it's also understandable that > most developers of PyPI clients wouldn't be aware of upcoming changes > that have only been discussed in detail on that list. I don't think the infra sig is the right host for such discussions and decisions. I'd suggest to use the distutils-sig and make Donald/Richard the PEP master for PyPI things, as they are maintaining it. > So, as Holger said, great work and thanks for your efforts, but good > communication does matter with these things. People don't like > surprises, even well intentioned ones :) We've had the CDN discussion for quite a while and I even setup a test CDN some months ago. No one ever mentioned the HTTP/1.0 problem and so it simply wasn't on the radar. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 34 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Tue May 28 16:23:15 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 10:23:15 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Message-ID: On May 28, 2013, at 8:20 AM, Donald Stufft wrote: > > On May 28, 2013, at 5:04 AM, Christian Theune wrote: > >> Hi, >> >> >> On 27. May2013, at 10:41 PM, Donald Stufft wrote: >>> Just to assure folks. I do consider Mirroring a first class citizen and an important feature. >> >> Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon. >> >> I agree with Holgers points. >> >> I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. >> >> My current client implementation is pretty simple and has had reliable semantics until now. >> >> A couple of things I noticed in the discussion that I'd like to point out: >> >> - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken. >> >> - Release files are replaced all the time. >> >> The semantics that I like to keep with the mirrors is this: >> >> When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X (but maybe also partial changes from X+n). >> >> I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. >> >> I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting. >> >> So - what's the next step that can happen ASAP? > > Options) > > 1) When mirroring retain N minutes worth of old serials and redo them. Mirroring is idempotent you can repeat it with no negative side effects. Conditional HTTP requests should also be supported to minimize the bandwidth. > 2) Wait a few seconds after fetching the change log to begin processing. > 3) Use front.python.org with the pypi.python.org HOST header with the caveat this is not guaranteed to be stable in the long term. > 4) ??? > Option 4: We add the expected hash of the simple page to the change log. Mirror clients can then assert their state consistent. Should also probably assert the file hashes that are in the simple index. > Of them 1) is more likely to give you the best resultshttp://mail.python.org/pipermail/distutils-sig/2013-May/020855.html the constraints of HTTP. All it takes is someone to run your mirroring script behind a caching proxy and pre-CDN you'd have the exact situation we have now. > > Mirroring is in a bad state because it comes (and has always) with absolutely no guarantees of consistency. You dismiss the issues of having serial n+1 changes, but that is a serious problem. If you fetch up to serial N of package1 which has the released version of 1.0, and then you fetch serial N+2 of package2 which has a hard requirement on package 1.1 (which was released in serial N+1) you now have packages that are not installable via your mirror because of inconsistent state. > > If someone comes up with a better option that doesn't require a large rearch of the storage code in PyPI I'm happy to review and deploy it. > >> >> Christian >> >> -- >> Christian Theune ? ct at gocept.com >> gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany >> http://gocept.com ? Tel +49 345 1229889-7 >> Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue May 28 16:36:20 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 May 2013 14:36:20 +0000 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: References: <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Message-ID: <20130528143620.GA7789@merlinux.eu> On Tue, May 28, 2013 at 10:23 -0400, Donald Stufft wrote: > On May 28, 2013, at 8:20 AM, Donald Stufft wrote: > > > > > On May 28, 2013, at 5:04 AM, Christian Theune wrote: > > > >> Hi, > >> > >> > >> On 27. May2013, at 10:41 PM, Donald Stufft wrote: > >>> Just to assure folks. I do consider Mirroring a first class citizen and an important feature. > >> > >> Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon. > >> > >> I agree with Holgers points. > >> > >> I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. > >> > >> My current client implementation is pretty simple and has had reliable semantics until now. > >> > >> A couple of things I noticed in the discussion that I'd like to point out: > >> > >> - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken. > >> > >> - Release files are replaced all the time. > >> > >> The semantics that I like to keep with the mirrors is this: > >> > >> When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X (but maybe also partial changes from X+n). > >> > >> I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. > >> > >> I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting. > >> > >> So - what's the next step that can happen ASAP? > > > > Options) > > > > 1) When mirroring retain N minutes worth of old serials and redo them. Mirroring is idempotent you can repeat it with no negative side effects. Conditional HTTP requests should also be supported to minimize the bandwidth. > > 2) Wait a few seconds after fetching the change log to begin processing. > > 3) Use front.python.org with the pypi.python.org HOST header with the caveat this is not guaranteed to be stable in the long term. > > 4) ??? > > > > Option 4: We add the expected hash of the simple page to the change log. Mirror clients can then assert their state consistent. > > Should also probably assert the file hashes that are in the simple index. yes, i also thought of option 4. Is that easy to implement on the side of pypi? If we checksum the simple-page, we need idem-potent generation of simple pages and ordering to begin with -- which is probably anyway a good idea. It doesn't need to be version-ordering, just some consistent ordering. As mentioned in the other mail, for the short-term i'd go for 3) once Noah and you confirm you are not going to kill it before we have settled on a new solution (maybe option 4). best, holger > > Of them 1) is more likely to give you the best resultshttp://mail.python.org/pipermail/distutils-sig/2013-May/020855.html the constraints of HTTP. All it takes is someone to run your mirroring script behind a caching proxy and pre-CDN you'd have the exact situation we have now. > > > > Mirroring is in a bad state because it comes (and has always) with absolutely no guarantees of consistency. You dismiss the issues of having serial n+1 changes, but that is a serious problem. If you fetch up to serial N of package1 which has the released version of 1.0, and then you fetch serial N+2 of package2 which has a hard requirement on package 1.1 (which was released in serial N+1) you now have packages that are not installable via your mirror because of inconsistent state. > > > > If someone comes up with a better option that doesn't require a large rearch of the storage code in PyPI I'm happy to review and deploy it. > > > >> > >> Christian > >> > >> -- > >> Christian Theune ? ct at gocept.com > >> gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany > >> http://gocept.com ? Tel +49 345 1229889-7 > >> Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations > > > > > > ----------------- > > Donald Stufft > > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Tue May 28 16:38:21 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 10:38:21 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <20130528143620.GA7789@merlinux.eu> References: <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> <20130528143620.GA7789@merlinux.eu> Message-ID: On May 28, 2013, at 10:36 AM, holger krekel wrote: > On Tue, May 28, 2013 at 10:23 -0400, Donald Stufft wrote: >> On May 28, 2013, at 8:20 AM, Donald Stufft wrote: >> >>> >>> On May 28, 2013, at 5:04 AM, Christian Theune wrote: >>> >>>> Hi, >>>> >>>> >>>> On 27. May2013, at 10:41 PM, Donald Stufft wrote: >>>>> Just to assure folks. I do consider Mirroring a first class citizen and an important feature. >>>> >>>> Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon. >>>> >>>> I agree with Holgers points. >>>> >>>> I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. >>>> >>>> My current client implementation is pretty simple and has had reliable semantics until now. >>>> >>>> A couple of things I noticed in the discussion that I'd like to point out: >>>> >>>> - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken. >>>> >>>> - Release files are replaced all the time. >>>> >>>> The semantics that I like to keep with the mirrors is this: >>>> >>>> When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X (but maybe also partial changes from X+n). >>>> >>>> I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. >>>> >>>> I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting. >>>> >>>> So - what's the next step that can happen ASAP? >>> >>> Options) >>> >>> 1) When mirroring retain N minutes worth of old serials and redo them. Mirroring is idempotent you can repeat it with no negative side effects. Conditional HTTP requests should also be supported to minimize the bandwidth. >>> 2) Wait a few seconds after fetching the change log to begin processing. >>> 3) Use front.python.org with the pypi.python.org HOST header with the caveat this is not guaranteed to be stable in the long term. >>> 4) ??? >> >> Option 4: We add the expected hash of the simple page to the change log. Mirror clients can then assert their state consistent. >> >> Should also probably assert the file hashes that are in the simple index. > > yes, i also thought of option 4. Is that easy to implement on the side of pypi? > If we checksum the simple-page, we need idem-potent generation of simple pages > and ordering to begin with -- which is probably anyway a good idea. > It doesn't need to be version-ordering, just some consistent ordering. > > As mentioned in the other mail, for the short-term i'd go for 3) once Noah > and you confirm you are not going to kill it before we have settled on > a new solution (maybe option 4). #3 is how fastly connects. > > best, > holger > > >>> Of them 1) is more likely to give you the best resultshttp://mail.python.org/pipermail/distutils-sig/2013-May/020855.html the constraints of HTTP. All it takes is someone to run your mirroring script behind a caching proxy and pre-CDN you'd have the exact situation we have now. >>> >>> Mirroring is in a bad state because it comes (and has always) with absolutely no guarantees of consistency. You dismiss the issues of having serial n+1 changes, but that is a serious problem. If you fetch up to serial N of package1 which has the released version of 1.0, and then you fetch serial N+2 of package2 which has a hard requirement on package 1.1 (which was released in serial N+1) you now have packages that are not installable via your mirror because of inconsistent state. >>> >>> If someone comes up with a better option that doesn't require a large rearch of the storage code in PyPI I'm happy to review and deploy it. >>> >>>> >>>> Christian >>>> >>>> -- >>>> Christian Theune ? ct at gocept.com >>>> gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany >>>> http://gocept.com ? Tel +49 345 1229889-7 >>>> Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations >>> >>> >>> ----------------- >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig > >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > From donald at stufft.io Tue May 28 16:39:22 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 10:39:22 -0400 Subject: [Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN) In-Reply-To: <20130528143620.GA7789@merlinux.eu> References: <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> <20130528143620.GA7789@merlinux.eu> Message-ID: On May 28, 2013, at 10:36 AM, holger krekel wrote: > On Tue, May 28, 2013 at 10:23 -0400, Donald Stufft wrote: >> On May 28, 2013, at 8:20 AM, Donald Stufft wrote: >> >>> >>> On May 28, 2013, at 5:04 AM, Christian Theune wrote: >>> >>>> Hi, >>>> >>>> >>>> On 27. May2013, at 10:41 PM, Donald Stufft wrote: >>>>> Just to assure folks. I do consider Mirroring a first class citizen and an important feature. >>>> >>>> Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon. >>>> >>>> I agree with Holgers points. >>>> >>>> I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. >>>> >>>> My current client implementation is pretty simple and has had reliable semantics until now. >>>> >>>> A couple of things I noticed in the discussion that I'd like to point out: >>>> >>>> - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken. >>>> >>>> - Release files are replaced all the time. >>>> >>>> The semantics that I like to keep with the mirrors is this: >>>> >>>> When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X (but maybe also partial changes from X+n). >>>> >>>> I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. >>>> >>>> I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting. >>>> >>>> So - what's the next step that can happen ASAP? >>> >>> Options) >>> >>> 1) When mirroring retain N minutes worth of old serials and redo them. Mirroring is idempotent you can repeat it with no negative side effects. Conditional HTTP requests should also be supported to minimize the bandwidth. >>> 2) Wait a few seconds after fetching the change log to begin processing. >>> 3) Use front.python.org with the pypi.python.org HOST header with the caveat this is not guaranteed to be stable in the long term. >>> 4) ??? >> >> Option 4: We add the expected hash of the simple page to the change log. Mirror clients can then assert their state consistent. >> >> Should also probably assert the file hashes that are in the simple index. > > yes, i also thought of option 4. Is that easy to implement on the side of pypi? > If we checksum the simple-page, we need idem-potent generation of simple pages > and ordering to begin with -- which is probably anyway a good idea. > It doesn't need to be version-ordering, just some consistent ordering. Check summing is easy yes. > > As mentioned in the other mail, for the short-term i'd go for 3) once Noah > and you confirm you are not going to kill it before we have settled on > a new solution (maybe option 4). > > best, > holger > > >>> Of them 1) is more likely to give you the best resultshttp://mail.python.org/pipermail/distutils-sig/2013-May/020855.html the constraints of HTTP. All it takes is someone to run your mirroring script behind a caching proxy and pre-CDN you'd have the exact situation we have now. >>> >>> Mirroring is in a bad state because it comes (and has always) with absolutely no guarantees of consistency. You dismiss the issues of having serial n+1 changes, but that is a serious problem. If you fetch up to serial N of package1 which has the released version of 1.0, and then you fetch serial N+2 of package2 which has a hard requirement on package 1.1 (which was released in serial N+1) you now have packages that are not installable via your mirror because of inconsistent state. >>> >>> If someone comes up with a better option that doesn't require a large rearch of the storage code in PyPI I'm happy to review and deploy it. >>> >>>> >>>> Christian >>>> >>>> -- >>>> Christian Theune ? ct at gocept.com >>>> gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany >>>> http://gocept.com ? Tel +49 345 1229889-7 >>>> Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations >>> >>> >>> ----------------- >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig > >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > From martin at v.loewis.de Tue May 28 17:49:30 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 May 2013 17:49:30 +0200 Subject: [Distutils] [Infrastructure] Good news everyone, PyPI is behind a CDN In-Reply-To: <51A4A799.6050603@egenix.com> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <877giki0fo.fsf@brainbot.com> <0C98B091-5943-4B53-9191-99AA9CED5972@coderanger.net> <87obbwgjy9.fsf@brainbot.com> <72F0B1D4-BDDF-4E08-BD2D-2094CFAED479@stufft.io> <4B83EF12-2B1D-4105-ACEA-41E47925B99F@stufft.io> <20130528065713.GL7789@merlinux.eu> <51A4A799.6050603@egenix.com> Message-ID: <51A4D20A.1080008@v.loewis.de> Am 28.05.13 14:48, schrieb M.-A. Lemburg: > We've had the CDN discussion for quite a while and I even setup > a test CDN some months ago. No one ever mentioned the HTTP/1.0 > problem and so it simply wasn't on the radar. On the other hand, the other problems *where* mentioned with respect to CDNs multiple times over the recent years, so this shouldn't have surprised anybody. Regards, Martin From martin at v.loewis.de Tue May 28 17:47:20 2013 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 May 2013 17:47:20 +0200 Subject: [Distutils] [Infrastructure] changelog / CDN inconsistency In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Message-ID: <51A4D188.8070206@v.loewis.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Mirroring is in a bad state because it comes (and has always) with > absolutely no guarantees of consistency. That is not true. There are no absolute guarantees, but certainly partial guarantees of consistency. It's a kind of "eventual consistency": if the releases are all older than the mirror frequency, the mirror will be consistent. > You dismiss the issues of having serial n+1 changes, but that is a > serious problem. If you fetch up to serial N of package1 which has > the released version of 1.0, and then you fetch serial N+2 of > package2 which has a hard requirement on package 1.1 (which was > released in serial N+1) you now have packages that are not > installable via your mirror because of inconsistent state. Sure, but that is only a temporary problem, with a inconsistency window of a few minutes in the worst case - and it only occurs if serials N, N+1, and N+2 all happen within 5 minutes (i.e. two releases of package1, and one release of package2). When the mirror script runs again, it will find that serial N+1 already happened, and fetch package1 and package2 again. > If someone comes up with a better option that doesn't require a > large rearch of the storage code in PyPI I'm happy to review and > deploy it. This could be fixed by having PyPI provide old versions of the simple page. It would not be possible to do so exactly currently. However, excluding releases newer than a given date would be possible, by inspecting the journal. Regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlGk0YgACgkQavBT8H2dyNImkQCffy1BKiYNxV71Bvtxpk+UAwPc j7wAn39wK7vMmERQhpSTfJ5iBPcP3wCr =yZBk -----END PGP SIGNATURE----- From martin at v.loewis.de Tue May 28 18:02:44 2013 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 28 May 2013 18:02:44 +0200 Subject: [Distutils] [Infrastructure] changelog / CDN inconsistency In-Reply-To: References: <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> <20130528143620.GA7789@merlinux.eu> Message-ID: <51A4D524.1070902@v.loewis.de> Am 28.05.13 16:39, schrieb Donald Stufft: > On May 28, 2013, at 10:36 AM, holger krekel wrote: >> yes, i also thought of option 4. Is that easy to implement on the side of pypi? >> If we checksum the simple-page, we need idem-potent generation of simple pages >> and ordering to begin with -- which is probably anyway a good idea. >> It doesn't need to be version-ordering, just some consistent ordering. > > Check summing is easy yes. And there is already a guarantee of a stable checksum for simple pages, because of the server signing of simple pages (which also computes a hash of the simple page already). Regards, Martin From martin at v.loewis.de Tue May 28 18:00:29 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 May 2013 18:00:29 +0200 Subject: [Distutils] [Infrastructure] changelog / CDN inconsistency In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Message-ID: <51A4D49D.6060408@v.loewis.de> Am 28.05.13 16:23, schrieb Donald Stufft: > Option 4: We add the expected hash of the simple page to the change log. > Mirror clients can then assert their state consistent. That would work. It would also cover the case where a new release happens while the mirroring is in progress. On the other hand, it's difficult to advise what to do if you find that the simple page does *not* match the most recent hashsum. You'ld have to wait a little bit, and hope that the CDN will eventually provide the current version. > Should also probably assert the file hashes that are in the simple index. Indeed, with the same limitation as above: if you find that the CDN gives the old version, you'll have to wait (or bypass the CDN). Regards, Martin From donald at stufft.io Tue May 28 18:06:43 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 12:06:43 -0400 Subject: [Distutils] [Infrastructure] changelog / CDN inconsistency In-Reply-To: <51A4D49D.6060408@v.loewis.de> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> <51A4D49D.6060408@v.loewis.de> Message-ID: <63D377EE-DA16-4F0E-AF78-D9A4481471D1@stufft.io> On May 28, 2013, at 12:00 PM, "Martin v. L?wis" wrote: > Am 28.05.13 16:23, schrieb Donald Stufft: >> Option 4: We add the expected hash of the simple page to the change log. >> Mirror clients can then assert their state consistent. > > That would work. It would also cover the case where a new release > happens while the mirroring is in progress. > > On the other hand, it's difficult to advise what to do if you find that > the simple page does *not* match the most recent hashsum. You'ld have to > wait a little bit, and hope that the CDN will eventually provide the > current version. > >> Should also probably assert the file hashes that are in the simple index. > > Indeed, with the same limitation as above: if you find that the CDN > gives the old version, you'll have to wait (or bypass the CDN). > > Regards, > Martin > > Immediately after committing the database transaction PyPI tells the CDN to purge it's cache for the packages that have been affected. Fastly advertises "instant" purging and in practice this means that the CDN will be serving the current version in less than a second after the database transaction has been commited. At certainly at the most a handful of seconds. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From martin at v.loewis.de Tue May 28 18:31:02 2013 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 May 2013 18:31:02 +0200 Subject: [Distutils] changelog / CDN inconsistency In-Reply-To: <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> Message-ID: <51A4DBC6.4010809@v.loewis.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 28.05.13 11:04, schrieb Christian Theune: > So - what's the next step that can happen ASAP? In addition to the changes Donald already did, I think it would be wise to restart mirroring at min(last_serial, last_mirroring - 1 minute). This will cause any simple pages to be re-downloaded that had been updated just before the last mirror run completed. If you aren't checking md5sums of files after download, you should (I always wanted to put this into pep381client). Then, if you re-download the simple page, you can skip files that you already have downloaded, and whose md5sum did not change. Regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlGk28UACgkQavBT8H2dyNKmSACffo+Nwa0R+csgRgm/5fJUsqUY Xm0AnjanJrexpu7Y/Rv0CJP76r6rdsS7 =oMsM -----END PGP SIGNATURE----- From martin at v.loewis.de Tue May 28 18:35:21 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 May 2013 18:35:21 +0200 Subject: [Distutils] pypi protocol In-Reply-To: References: Message-ID: <51A4DCC9.7050003@v.loewis.de> Am 26.05.13 22:08, schrieb Jonas Geiregat: > I ended up reading pypiserver's source code to find out the internals. Notice that this is the wrong source code. The real PyPI source code is in https://bitbucket.org/pypa/pypi/src Regards, Martin From erik.m.bray at gmail.com Tue May 28 20:07:43 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 28 May 2013 14:07:43 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: On Mon, May 27, 2013 at 7:36 AM, Nick Coghlan wrote: > After preliminary reviews by Donald and Daniel, I have now pushed the > first complete draft of the JSON-based metadata 2.0 proposal to > python.org > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ > > With the rationale and commentary, they're over 3000 lines between > them, so I'm not attaching them here. > > The rationale for many of the changes is at the end of each PEP, along > with some comments on features that I have either rejected or > deliberately chosen to defer to the next revision of the metadata (at > the earliest). > > Those with BitBucket accounts may also comment inline on the drafts here: > > PEP 426: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0426.txt?at=default > PEP 440: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0440.txt?at=default This is looking fantastic so far--thanks to Nick, Daniel, and Donald for their continued work on this. For now I just have a handful of minor notes on the latest draft of PEP 426: Typos: Under "Essential dependency resolution metadata" the "may_require" and related metadata keywords are spelled with hyphens instead of underscores. Under "Metabuild system" in the first example I think "some_test_harness.metabuild_hook" was meant to read "some_test_harness:metabuild_hook" Under "Development, build and deployment dependencies": "allow" -> "allows" Under "Support for metabuild hooks": "by allows projects" -> "by allowing projects" Comment: I'm not sure if this PEP is the best place for this, but I wonder if the description of the "Keywords" format could provide some clarification on how that field should be formatted in older metadata versions (specifically when including version 1.x metadata for backwards compatibility). In the past its format has never been specified. Some tools treat it as a space-separated fields. Others have treated it as a comma-separated field. Sometimes one or the other depending on whether commas are present. It's a very annoying field. Erik From erik.m.bray at gmail.com Tue May 28 20:21:06 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 28 May 2013 14:21:06 -0400 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> Message-ID: On Mon, May 27, 2013 at 1:19 AM, Lennart Regebro wrote: > On Sun, May 26, 2013 at 7:34 PM, Noah Kantrowitz wrote: >> >> >> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. > > This is going to spell disaster for the coffee industry, as you no > longer have to take a coffee break when re-running a buildout. > > Thanks! I always test pip installation from PyPI "just in case" after uploading a new package, so the new cache delay still leaves some time for a coffee break (until Daniel gets the cache invalidation integrated :/). But yes, so many hoorays for this \o/ From donald at stufft.io Tue May 28 20:23:01 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 14:23:01 -0400 Subject: [Distutils] Good news everyone, PyPI is behind a CDN In-Reply-To: References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> Message-ID: <9B448FF5-0360-4555-9780-84487372B41C@stufft.io> On May 28, 2013, at 2:21 PM, Erik Bray wrote: > On Mon, May 27, 2013 at 1:19 AM, Lennart Regebro wrote: >> On Sun, May 26, 2013 at 7:34 PM, Noah Kantrowitz wrote: >>> >>> >>> but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. >> >> This is going to spell disaster for the coffee industry, as you no >> longer have to take a coffee break when re-running a buildout. >> >> Thanks! > > I always test pip installation from PyPI "just in case" after > uploading a new package, so the new cache delay still leaves some time > for a coffee break (until Daniel gets the cache invalidation > integrated :/). But yes, so many hoorays for this \o/ I already enabled Cache Invalidation. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Tue May 28 22:43:46 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 28 May 2013 16:43:46 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: On Tue, May 28, 2013 at 2:07 PM, Erik Bray wrote: > On Mon, May 27, 2013 at 7:36 AM, Nick Coghlan wrote: >> After preliminary reviews by Donald and Daniel, I have now pushed the >> first complete draft of the JSON-based metadata 2.0 proposal to >> python.org >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ >> PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ >> >> With the rationale and commentary, they're over 3000 lines between >> them, so I'm not attaching them here. >> >> The rationale for many of the changes is at the end of each PEP, along >> with some comments on features that I have either rejected or >> deliberately chosen to defer to the next revision of the metadata (at >> the earliest). >> >> Those with BitBucket accounts may also comment inline on the drafts here: >> >> PEP 426: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0426.txt?at=default >> PEP 440: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0440.txt?at=default > > This is looking fantastic so far--thanks to Nick, Daniel, and Donald > for their continued work on this. For now I just have a handful of > minor notes on the latest draft of PEP 426: > > Typos: > > Under "Essential dependency resolution metadata" the "may_require" and > related metadata keywords are spelled with hyphens instead of > underscores. > > Under "Metabuild system" in the first example I think > "some_test_harness.metabuild_hook" was meant to read > "some_test_harness:metabuild_hook" > > > Under "Development, build and deployment dependencies": "allow" -> "allows" > > Under "Support for metabuild hooks": "by allows projects" -> "by > allowing projects" > > Comment: > > I'm not sure if this PEP is the best place for this, but I wonder if > the description of the "Keywords" format could provide some > clarification on how that field should be formatted in older metadata > versions (specifically when including version 1.x metadata for > backwards compatibility). In the past its format has never been > specified. Some tools treat it as a space-separated fields. Others > have treated it as a comma-separated field. Sometimes one or the > other depending on whether commas are present. It's a very annoying > field. I suggest treating it as a space-separated field for converting from 2.0 to 1.0. To convert from 1.0 to 2.0 you should just split on "not a letter" or if you are feeling ambitious "not some larger set of characters, probably resembling the identifier or package name rules". From ralf at systemexit.de Wed May 29 01:32:41 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Wed, 29 May 2013 01:32:41 +0200 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server Message-ID: <87ip227k9y.fsf@myhost.lan> Hi, I've just uploaded pypiserver 1.1.1 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. https://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver Changes in this version ----------------------- - add 'overwrite' option to allow overwriting existing package files (default: false) - show names with hyphens instead of underscores on the "/simple" listing - make the standalone version work with jython 2.5.3 - upgrade waitress to 0.8.5 in the standalone version - workaround broken xmlrpc api on pypi.python.org by using HTTPS -- Cheers Ralf From ncoghlan at gmail.com Wed May 29 03:00:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 May 2013 11:00:52 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: On Mon, May 27, 2013 at 9:36 PM, Nick Coghlan wrote: > After preliminary reviews by Donald and Daniel, I have now pushed the > first complete draft of the JSON-based metadata 2.0 proposal to > python.org > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ Based on some offline feedback from Daniel, I'm going to change the current "type" field in the contact metadata to "role". The name of the default role will change from "individual" to "contributor", and projects will be given freedom to define their own roles beyond the predefined ones. (We're actually stealing this from the way contact metadata works in PHP's composer). Cheers, Nick. From donald at stufft.io Wed May 29 03:04:23 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 May 2013 21:04:23 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: <73455760-C9EB-4752-B8FC-BDD4F13CD352@stufft.io> On May 28, 2013, at 9:00 PM, Nick Coghlan wrote: > On Mon, May 27, 2013 at 9:36 PM, Nick Coghlan wrote: >> After preliminary reviews by Donald and Daniel, I have now pushed the >> first complete draft of the JSON-based metadata 2.0 proposal to >> python.org >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ >> PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ > > Based on some offline feedback from Daniel, I'm going to change the > current "type" field in the contact metadata to "role". The name of > the default role will change from "individual" to "contributor", and > projects will be given freedom to define their own roles beyond the > predefined ones. (We're actually stealing this from the way contact > metadata works in PHP's composer). > > Cheers, > Nick. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Please define what the valid values for the role field are when you include it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed May 29 03:16:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 May 2013 11:16:28 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <73455760-C9EB-4752-B8FC-BDD4F13CD352@stufft.io> References: <73455760-C9EB-4752-B8FC-BDD4F13CD352@stufft.io> Message-ID: On Wed, May 29, 2013 at 11:04 AM, Donald Stufft wrote: > > On May 28, 2013, at 9:00 PM, Nick Coghlan wrote: > > On Mon, May 27, 2013 at 9:36 PM, Nick Coghlan wrote: > > After preliminary reviews by Donald and Daniel, I have now pushed the > first complete draft of the JSON-based metadata 2.0 proposal to > python.org > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ > > > Based on some offline feedback from Daniel, I'm going to change the > current "type" field in the contact metadata to "role". The name of > the default role will change from "individual" to "contributor", and > projects will be given freedom to define their own roles beyond the > predefined ones. (We're actually stealing this from the way contact > metadata works in PHP's composer). Hmm, I may actually drop the extensibility idea - it makes the tooling harder without providing a significant benefit. So just the name changes for the field and the default value. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed May 29 03:14:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 May 2013 11:14:51 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <14C2D779-8D99-4036-948A-7BF367A1DCB6@stufft.io> References: <06FE1104-014B-4EA1-9CAF-6BEC0FAFD4F1@mac.com> <14C2D779-8D99-4036-948A-7BF367A1DCB6@stufft.io> Message-ID: On Tue, May 28, 2013 at 7:28 AM, Donald Stufft wrote: > On May 27, 2013, at 10:44 AM, Ronald Oussoren > wrote: > The versioning spec mentions that distribution tools may refuse to publish > distributions that pin the versions of dependencies. I understand why this > is needed, and agree in general, but have a usecase that I don't know how to > express without pinning. > > In particular, PyObjC consists of a number of distributions (pyobjc-core, > pyobjc-framework-Cocoa, ...) and an umbrella package (pyobjc) what depends > on the various distributions to make it easier to install all of PyObjC. The > umbrella package currently pins the versions of subpackages to ensure that > "pip install pyobjc==2.5.1" installs exactly that version of the entire > project. When I'd use the "compatible release" specifier I can no longer > easily ensure that users can install an exact version of the entire project, > other than by hacking the system: specify a compatible version with an > additional level that isn't used by the project (for example ~=2.5.2.0). > What is the correct way to create an umberella project without getting > yelled at by distribution tools? > > > It's unlikely PyPI will get more than a warning for ``==``, `is` comparisons > might be disallowed? Not sure. I think Ronald's example of publishing metadistributions that pin particular versions of subdistributions is a valid one (I do exactly the same thing myself with RPM, it just didn't occur to me as a use case while updating the PEPs), so I need to reconsider some of the index server restrictions currently proposed in the PEPs. However, I'd also still like to not-so-gently steer users away from overly restrictive dependencies in the general case. This is a case where in a *technical* sense there's no difference between "We are making these distributions we maintain easier to install all at once" and "Our distribution needs a compatible version of this other distribution in order to work", but *semantically* they're two quite different operations. So, what do people think of the idea of a new top level "distributes" field? Syntax identical to "requires", but *semantically* distinguished in that version pinning in "distributes" would be not only allowed, but encouraged. A metapackage like PyObjC would then have just entries in the "distributes" field, and no direct dependencies of its own. Thoughts? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed May 29 05:12:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 May 2013 13:12:41 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: A couple of significant upcoming changes: "build label" will be renamed as "source label" (since it refers to the common unbuilt source rather than a specific build) "version URL" will be renamed as "source URL" (same rationale) The current names are ambiguous as to whether they refer to the source code for the version or can be used to refer to built versions. Since they're specifically for source references (you need to add at least PEP 425 compatibility tags to construct a built reference), it makes sense to change the names. A more minor change is that the "organization" type/role for contacts will go away. Organization will be able to have any of the defined roles (author, maintainer, contributor) and if we later decide we need a programmatic means to distinguish abstract organisations from flesh and blood humans we can consider adding a new mechanism. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jonas at geiregat.org Wed May 29 09:10:48 2013 From: jonas at geiregat.org (Jonas Geiregat) Date: Wed, 29 May 2013 09:10:48 +0200 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server In-Reply-To: <87ip227k9y.fsf@myhost.lan> References: <87ip227k9y.fsf@myhost.lan> Message-ID: <0CE118D8-24C3-4E40-B12D-AAA047AA5150@geiregat.org> Hello Ralf, I already started working on your code and using it a s a base for my pypiserver project. Thanks again, great piece of minimal work! Jonas. On 29 May 2013, at 01:32, Ralf Schmitt wrote: > Hi, > > I've just uploaded pypiserver 1.1.1 to the python package index. > > pypiserver is a minimal PyPI compatible server. It can be used to serve > a set of packages and eggs to easy_install or pip. > > pypiserver is easy to install (i.e. just 'pip install pypiserver'). It > doesn't have any external dependencies. > > https://pypi.python.org/pypi/pypiserver/ should contain enough > information to easily get you started running your own PyPI server in a > few minutes. > > The code is available on github: https://github.com/schmir/pypiserver > > Changes in this version > ----------------------- > - add 'overwrite' option to allow overwriting existing package > files (default: false) > - show names with hyphens instead of underscores on the "/simple" > listing > - make the standalone version work with jython 2.5.3 > - upgrade waitress to 0.8.5 in the standalone version > - workaround broken xmlrpc api on pypi.python.org by using HTTPS > > -- > Cheers > Ralf > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From marius at pov.lt Wed May 29 10:19:43 2013 From: marius at pov.lt (Marius Gedminas) Date: Wed, 29 May 2013 11:19:43 +0300 Subject: [Distutils] Buildout can't talk to PyPI any more? Message-ID: <20130529081943.GB30835@fridge.pov.lt> Could it be that recent PyPI CDN changes broke zc.buildout somehow? Witness this: $ wget http://downloads.buildout.org/2/bootstrap.py $ cat buildout.cfg [buildout] parts = zodbbrowser [zodbbrowser] recipe = zc.recipe.egg $ python bootstrap.py $ bin/buildout Error: Couldn't find a distribution for 'zodbbrowser'. Meanwhile both $ virtualenv env && env/bin/pip install zodbbrowser and $ virtualenv env && env/bin/easy_install zodbbrowser work fine. Marius Gedminas -- Those who can't write, write manuals. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From lists at zopyx.com Wed May 29 10:22:29 2013 From: lists at zopyx.com (Andreas Jung) Date: Wed, 29 May 2013 10:22:29 +0200 Subject: [Distutils] PyPI CDN issues Message-ID: <51A5BAC5.6090600@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, after switching to the new PyPI CDN infrastructure we have seen some odd behavior with buildout fetching two hundred packages or so. Packages that are a located on PyPI have not been available to the CDN or only after some minutes. Note that these packages have not been fresh releases but older stuff. Is this a known issues or explainable behavior of the CDN? Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJRpbrFAAoJEADcfz7u4AZjtDQLvivb5VjrTDe8y9/8mhiCnEPm rJ6Vk9YGsHTCd5zKLWCycmDCB4aFJFDN9jZ+6Hz8xizXmH+Jpy8rPM+f1p+I5szw tWRjeT79TwEwaoR3TUj54tmSLxZp+/6+DMxWG741RXE5BGKPqzi8qENSXBTbnL7P r3TionD6bEV0J5BCb+9MCt0Ml/4u4L0z4nyEVifAFq5Z9FbB7ja2RUKlWQxHSw+1 6ojLpOMqaEzQxx6ANPg4KXSbetLSiY1r1Qq6EhgqtCjudtgn8zUqEKgzr3XJ52Bi D2aVd/imj0S48cu4TJkg3kF7Z+SVd0XuxXpoYm6N9op9BJVWNcIucB49sujDknrZ sdRI9Nl4DZiL+nfk2GT5VqMbG+V5+WwwQw3V9ew5jLz5VcjGi2IbnGPjDBRH9UWc hemzGJlyo/JdsKmqpY30repSi/1Z7foZPWtANyIOcGNNfTuiZc3hRyZVrQyRpcI4 lwTei5swGj271umNIZIs1ZiMfInM0vo= =E8Ij -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 353 bytes Desc: not available URL: From ralf at systemexit.de Wed May 29 10:33:35 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Wed, 29 May 2013 10:33:35 +0200 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server In-Reply-To: <0CE118D8-24C3-4E40-B12D-AAA047AA5150@geiregat.org> (Jonas Geiregat's message of "Wed, 29 May 2013 09:10:48 +0200") References: <87ip227k9y.fsf@myhost.lan> <0CE118D8-24C3-4E40-B12D-AAA047AA5150@geiregat.org> Message-ID: <87txlmdw2o.fsf@myhost.lan> Jonas Geiregat writes: > Hello Ralf, > > I already started working on your code and using it a s a base for my pypiserver project. > > Thanks again, great piece of minimal work! Thanks! What features do you plan to implement? Maybe some of that functionality is already available in other implementations. http://bitofcheese.blogspot.de/2013/05/local-pypi-options.html contains a nice rundown of alternative implementations. I've tested proxypypi and it's quite nice if you need a caching proxy. I assume devpi-server is also quite good given the fact that it's author produces really nice work. -- Cheers Ralf From regebro at gmail.com Wed May 29 11:03:39 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 29 May 2013 11:03:39 +0200 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: <20130529081943.GB30835@fridge.pov.lt> References: <20130529081943.GB30835@fridge.pov.lt> Message-ID: On Wed, May 29, 2013 at 10:19 AM, Marius Gedminas wrote: > Could it be that recent PyPI CDN changes broke zc.buildout somehow? > Witness this: > > $ wget http://downloads.buildout.org/2/bootstrap.py > > $ cat buildout.cfg > [buildout] > parts = zodbbrowser > [zodbbrowser] > recipe = zc.recipe.egg > > $ python bootstrap.py > > $ bin/buildout > Error: Couldn't find a distribution for 'zodbbrowser'. Works for me. From jonas at geiregat.org Wed May 29 11:09:26 2013 From: jonas at geiregat.org (Jonas Geiregat) Date: Wed, 29 May 2013 11:09:26 +0200 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server In-Reply-To: <87txlmdw2o.fsf@myhost.lan> References: <87ip227k9y.fsf@myhost.lan> <0CE118D8-24C3-4E40-B12D-AAA047AA5150@geiregat.org> <87txlmdw2o.fsf@myhost.lan> Message-ID: On 29 May 2013, at 10:33, Ralf Schmitt wrote: > Jonas Geiregat writes: > >> Hello Ralf, >> >> I already started working on your code and using it a s a base for my pypiserver project. >> >> Thanks again, great piece of minimal work! > > Thanks! What features do you plan to implement? I will be using your code to transform it into a Nexus alike system. First I added jinja2 and sqlalchemy , that's almost finished. Next I want to add a decent web interface, probably based on bootstrap. From marius at pov.lt Wed May 29 11:18:23 2013 From: marius at pov.lt (Marius Gedminas) Date: Wed, 29 May 2013 12:18:23 +0300 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: References: <20130529081943.GB30835@fridge.pov.lt> Message-ID: <20130529091823.GA7606@fridge.pov.lt> On Wed, May 29, 2013 at 11:03:39AM +0200, Lennart Regebro wrote: > On Wed, May 29, 2013 at 10:19 AM, Marius Gedminas wrote: > > Could it be that recent PyPI CDN changes broke zc.buildout somehow? > > Witness this: > > > > $ wget http://downloads.buildout.org/2/bootstrap.py > > > > $ cat buildout.cfg > > [buildout] > > parts = zodbbrowser > > [zodbbrowser] > > recipe = zc.recipe.egg > > > > $ python bootstrap.py > > > > $ bin/buildout > > Error: Couldn't find a distribution for 'zodbbrowser'. > > Works for me. Works for me too, now. Interesting detail: when buildout was failing, and I tried curl http://pypi.python.org/simple/zodbbrowser/ all I got was a bunch of binary garbage. To be fair, curl -I showed Content-encoding: gzip, and piping curl to zcat showed the HTML just fine. But the interesting thing is that when I try this now, I get raw HTML. Can it be that buildout doesn't handle Content-Encoding: gzip? I was also looking at raw HTTP requests with ngrep, and saw that buildout issued something like GET /simple/zodbbrowser HTTP/forgotwhichversionsadly Host: ... Accept-Encoding: identity User-Agent: blah blah distribute 0.6.44 blah blah Python/urllib That "Accept-Encoding" header kind of hints that distribute can't handle gzipped index pages. So maybe Varnish or nginx or something on the CDN side was misconfigured and ignored the Accept-Encoding? But why would pip/easy_install work then -- they use distribute too!? (Or was I accidentally using a virtualenv that had setuptools instead of distribute? Python packaging aargh.) Another possibly interesting detail: I made a release of zodbbrowser 0.11.0 this morning. Buildout was unable to get it right after that, and this condition lasted for several hours (from 07:40 UTC until some point between 08:20 and 09:00 UTC). The Zope Windows buildbot had a couple of similar intermittent errors (buildout was unable to find a distribution of some package) that went away when the build job was retried after 24 hours. Could it be that something about the PyPI CDN causes new package releases to be gzipped for an hour or so, and then revert to plain HTML? I'm tempted to do a test, something like curl -I http://pypi.python.org/simple/somepackage/ cd ~/src/somepackage && fullrelease # zest.releaser FTW curl -I http://pypi.python.org/simple/somepackage/ if I can find some package ready for a new release. Marius Gedminas -- We'll show a small example here with one user calling another. (Under international treaties describing technical papers these users must be called "Alice" and "Bob".) -- Anthony Baxter -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From donald at stufft.io Wed May 29 11:31:54 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 05:31:54 -0400 Subject: [Distutils] PyPI CDN issues In-Reply-To: <51A5BAC5.6090600@zopyx.com> References: <51A5BAC5.6090600@zopyx.com> Message-ID: <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> On May 29, 2013, at 4:22 AM, Andreas Jung wrote: > Hi there, > > after switching to the new PyPI CDN infrastructure we have seen some > odd behavior with buildout fetching two hundred packages or so. > Packages that are a located on PyPI have not been available to the CDN > or only after some minutes. Note that these packages have not been fresh > releases but older stuff. Is this a known issues or explainable behavior > of the CDN? > > Andreas > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig This behavior shouldn't happen. Is it consistently happening? Especially for older releases the behavior should be transparent as far as finding/downloading packages goes. What error message are you getting when these packages are not available? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 29 11:39:08 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 05:39:08 -0400 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: <20130529091823.GA7606@fridge.pov.lt> References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> Message-ID: <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> On May 29, 2013, at 5:18 AM, Marius Gedminas wrote: > On Wed, May 29, 2013 at 11:03:39AM +0200, Lennart Regebro wrote: >> On Wed, May 29, 2013 at 10:19 AM, Marius Gedminas wrote: >>> Could it be that recent PyPI CDN changes broke zc.buildout somehow? >>> Witness this: >>> >>> $ wget http://downloads.buildout.org/2/bootstrap.py >>> >>> $ cat buildout.cfg >>> [buildout] >>> parts = zodbbrowser >>> [zodbbrowser] >>> recipe = zc.recipe.egg >>> >>> $ python bootstrap.py >>> >>> $ bin/buildout >>> Error: Couldn't find a distribution for 'zodbbrowser'. >> >> Works for me. > > Works for me too, now. > > > Interesting detail: when buildout was failing, and I tried > > curl http://pypi.python.org/simple/zodbbrowser/ > > all I got was a bunch of binary garbage. To be fair, curl -I showed > Content-encoding: gzip, and piping curl to zcat showed the HTML just > fine. But the interesting thing is that when I try this now, I get raw > HTML. > > Can it be that buildout doesn't handle Content-Encoding: gzip? > > I was also looking at raw HTTP requests with ngrep, and saw that > buildout issued something like > > GET /simple/zodbbrowser HTTP/forgotwhichversionsadly > Host: ... > Accept-Encoding: identity > User-Agent: blah blah distribute 0.6.44 blah blah Python/urllib > > That "Accept-Encoding" header kind of hints that distribute can't handle > gzipped index pages. So maybe Varnish or nginx or something on the CDN > side was misconfigured and ignored the Accept-Encoding? But why would > pip/easy_install work then -- they use distribute too!? (Or was I > accidentally using a virtualenv that had setuptools instead of > distribute? Python packaging aargh.) > > > Another possibly interesting detail: I made a release of zodbbrowser > 0.11.0 this morning. Buildout was unable to get it right after that, > and this condition lasted for several hours (from 07:40 UTC until some > point between 08:20 and 09:00 UTC). > > The Zope Windows buildbot had a couple of similar intermittent errors > (buildout was unable to find a distribution of some package) that went > away when the build job was retried after 24 hours. Could it be that > something about the PyPI CDN causes new package releases to be gzipped > for an hour or so, and then revert to plain HTML? > > I'm tempted to do a test, something like > > curl -I http://pypi.python.org/simple/somepackage/ > cd ~/src/somepackage && fullrelease # zest.releaser FTW > curl -I http://pypi.python.org/simple/somepackage/ > > if I can find some package ready for a new release. > > Marius Gedminas > -- > We'll show a small example here with one user calling another. (Under > international treaties describing technical papers these users must be called > "Alice" and "Bob".) > -- Anthony Baxter > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig The cache timeout is an hour unless modifications are made to the package which triggers a cache invalidation. I bet there is a missing Vary for the encoding. I'll check and get back to you. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 29 11:43:31 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 05:43:31 -0400 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> Message-ID: <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> On May 29, 2013, at 5:39 AM, Donald Stufft wrote: > > On May 29, 2013, at 5:18 AM, Marius Gedminas wrote: > >> On Wed, May 29, 2013 at 11:03:39AM +0200, Lennart Regebro wrote: >>> On Wed, May 29, 2013 at 10:19 AM, Marius Gedminas wrote: >>>> Could it be that recent PyPI CDN changes broke zc.buildout somehow? >>>> Witness this: >>>> >>>> $ wget http://downloads.buildout.org/2/bootstrap.py >>>> >>>> $ cat buildout.cfg >>>> [buildout] >>>> parts = zodbbrowser >>>> [zodbbrowser] >>>> recipe = zc.recipe.egg >>>> >>>> $ python bootstrap.py >>>> >>>> $ bin/buildout >>>> Error: Couldn't find a distribution for 'zodbbrowser'. >>> >>> Works for me. >> >> Works for me too, now. >> >> >> Interesting detail: when buildout was failing, and I tried >> >> curl http://pypi.python.org/simple/zodbbrowser/ >> >> all I got was a bunch of binary garbage. To be fair, curl -I showed >> Content-encoding: gzip, and piping curl to zcat showed the HTML just >> fine. But the interesting thing is that when I try this now, I get raw >> HTML. >> >> Can it be that buildout doesn't handle Content-Encoding: gzip? >> >> I was also looking at raw HTTP requests with ngrep, and saw that >> buildout issued something like >> >> GET /simple/zodbbrowser HTTP/forgotwhichversionsadly >> Host: ... >> Accept-Encoding: identity >> User-Agent: blah blah distribute 0.6.44 blah blah Python/urllib >> >> That "Accept-Encoding" header kind of hints that distribute can't handle >> gzipped index pages. So maybe Varnish or nginx or something on the CDN >> side was misconfigured and ignored the Accept-Encoding? But why would >> pip/easy_install work then -- they use distribute too!? (Or was I >> accidentally using a virtualenv that had setuptools instead of >> distribute? Python packaging aargh.) >> >> >> Another possibly interesting detail: I made a release of zodbbrowser >> 0.11.0 this morning. Buildout was unable to get it right after that, >> and this condition lasted for several hours (from 07:40 UTC until some >> point between 08:20 and 09:00 UTC). >> >> The Zope Windows buildbot had a couple of similar intermittent errors >> (buildout was unable to find a distribution of some package) that went >> away when the build job was retried after 24 hours. Could it be that >> something about the PyPI CDN causes new package releases to be gzipped >> for an hour or so, and then revert to plain HTML? >> >> I'm tempted to do a test, something like >> >> curl -I http://pypi.python.org/simple/somepackage/ >> cd ~/src/somepackage && fullrelease # zest.releaser FTW >> curl -I http://pypi.python.org/simple/somepackage/ >> >> if I can find some package ready for a new release. >> >> Marius Gedminas >> -- >> We'll show a small example here with one user calling another. (Under >> international treaties describing technical papers these users must be called >> "Alice" and "Bob".) >> -- Anthony Baxter >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > The cache timeout is an hour unless modifications are made to the package which triggers a cache invalidation. > > I bet there is a missing Vary for the encoding. I'll check and get back to you. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig There was indeed no Vary headers, I've turned them on and purged the cache. Please let me know if this behavior doesn't go away. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 29 11:47:35 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 05:47:35 -0400 Subject: [Distutils] PyPI CDN issues In-Reply-To: <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> Message-ID: On May 29, 2013, at 5:31 AM, Donald Stufft wrote: > > On May 29, 2013, at 4:22 AM, Andreas Jung wrote: > >> Hi there, >> >> after switching to the new PyPI CDN infrastructure we have seen some >> odd behavior with buildout fetching two hundred packages or so. >> Packages that are a located on PyPI have not been available to the CDN >> or only after some minutes. Note that these packages have not been fresh >> releases but older stuff. Is this a known issues or explainable behavior >> of the CDN? >> >> Andreas >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > This behavior shouldn't happen. Is it consistently happening? Especially for older releases the behavior should be transparent as far as finding/downloading packages goes. > > What error message are you getting when these packages are not available? > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig There was an issue with a missing Vary: Accept-Encoding and gzip pages getting cached and served even for non gzip enabled clients. I've fixed this and purged the cache. See if this fixes your issue. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From m.van.rees at zestsoftware.nl Wed May 29 12:18:52 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 29 May 2013 12:18:52 +0200 Subject: [Distutils] PyPI CDN issues In-Reply-To: References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> Message-ID: Op 29-05-13 11:47, Donald Stufft schreef: > There was an issue with a missing Vary: Accept-Encoding and gzip pages > getting cached and served even for non gzip enabled clients. I've fixed > this and purged the cache. See if this fixes your issue. I have problems on and off. I have created a virtualenv with python 2.7 and try to install zc.buildout. 1. easy_install first fails: mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install 'zc.buildout<2dev' Searching for zc.buildout<2dev Reading http://pypi.python.org/simple/zc.buildout/ No local packages or download links found for zc.buildout<2dev error: Could not find suitable distribution for Requirement.parse('zc.buildout<2dev') 2. Pip now works: mauritsvanrees at procyon:foo (git:master?) $ bin/pip install 'zc.buildout<2dev' Downloading/unpacking zc.buildout<2dev Downloading zc.buildout-1.7.1.tar.gz (288kB): 288kB downloaded Running setup.py egg_info for package zc.buildout Requirement already satisfied (use --upgrade to upgrade): distribute in ./lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg (from zc.buildout<2dev) Installing collected packages: zc.buildout Running setup.py install for zc.buildout Skipping installation of /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/lib/python2.7/site-packages/zc/__init__.py (namespace package) Installing /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/lib/python2.7/site-packages/zc.buildout-1.7.1-py2.7-nspkg.pth Installing buildout script to /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/bin Successfully installed zc.buildout Cleaning up... 3. Uninstall it again: mauritsvanrees at procyon:foo (git:master?) $ bin/pip uninstall zc.buildout Uninstalling zc.buildout: ... Successfully uninstalled zc.buildout 4. Give easy_install another failed chance: mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install 'zc.buildout<2dev' Searching for zc.buildout<2dev Reading http://pypi.python.org/simple/zc.buildout/ No local packages or download links found for zc.buildout<2dev error: Could not find suitable distribution for Requirement.parse('zc.buildout<2dev') mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install 'zc.buildout==1.7.1' Searching for zc.buildout==1.7.1 Reading http://pypi.python.org/simple/zc.buildout/ error: Connection reset by peer 5. easy_install explicitly 1.7.1, now it works: mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install zc.buildout==1.7.1 Searching for zc.buildout==1.7.1 Reading http://pypi.python.org/simple/zc.buildout/ Reading http://buildout.org Best match: zc.buildout 1.7.1 ... Processing dependencies for zc.buildout==1.7.1 Finished processing dependencies for zc.buildout==1.7.1 6. Uninstall it again: mauritsvanrees at procyon:foo (git:master?) $ bin/pip uninstall zc.buildout Uninstalling zc.buildout: /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/bin/buildout /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/lib/python2.7/site-packages/zc.buildout-1.7.1-py2.7.egg Proceed (y/n)? y Successfully uninstalled zc.buildout 6. easy_install the same 1.7.1 again, now it fails: mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install zc.buildout==1.7.1 Searching for zc.buildout==1.7.1 Reading http://pypi.python.org/simple/zc.buildout/ No local packages or download links found for zc.buildout==1.7.1 error: Could not find suitable distribution for Requirement.parse('zc.buildout==1.7.1') So: looks random to me. pip seems to do it good consistently, but that may be luck. Bootstrapping a buildout also fails, which is why I investigated: $ curl -O http://downloads.buildout.org/1/bootstrap.py $ python2.7 bootstrap.py No local packages or download links found for zc.buildout<2dev error: Could not find suitable distribution for Requirement.parse('zc.buildout<2dev') An error occurred when trying to install zc.buildout. Look above this message for any errors that were output by easy_install. Same with latest version: $ curl -O http://downloads.buildout.org/2/bootstrap.py $ python2.7 bootstrap.py No local packages or download links found for zc.buildout error: Could not find suitable distribution for Requirement.parse('zc.buildout') Traceback (most recent call last): File "bootstrap.py", line 148, in repr(cmd)[1:-1]) Exception: ('Failed to execute command:\n%s', "'/Users/mauritsvanrees/py/bin/python2.7', '-c', 'from setuptools.command.easy_install import main; main()', '-mZqNxd', '/var/folders/b3/4fkrg4_17ms0pwmwh0xy41_w0000gn/T/tmpSgaQlF', 'zc.buildout'") If you do 'python2.7 bootstrap.py -h' you will see some ways around this by pointing to a different download location. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From donald at stufft.io Wed May 29 12:34:39 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 06:34:39 -0400 Subject: [Distutils] PyPI CDN issues In-Reply-To: References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> Message-ID: <7048D244-0AD3-44C1-BA7D-E00B45E09B64@stufft.io> On May 29, 2013, at 6:18 AM, Maurits van Rees wrote: > Op 29-05-13 11:47, Donald Stufft schreef: >> There was an issue with a missing Vary: Accept-Encoding and gzip pages >> getting cached and served even for non gzip enabled clients. I've fixed >> this and purged the cache. See if this fixes your issue. > > I have problems on and off. > > I have created a virtualenv with python 2.7 and try to install zc.buildout. > > 1. easy_install first fails: > > mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install 'zc.buildout<2dev' > Searching for zc.buildout<2dev > Reading http://pypi.python.org/simple/zc.buildout/ > No local packages or download links found for zc.buildout<2dev > error: Could not find suitable distribution for Requirement.parse('zc.buildout<2dev') > > 2. Pip now works: > > mauritsvanrees at procyon:foo (git:master?) $ bin/pip install 'zc.buildout<2dev' > Downloading/unpacking zc.buildout<2dev > Downloading zc.buildout-1.7.1.tar.gz (288kB): 288kB downloaded > Running setup.py egg_info for package zc.buildout > > Requirement already satisfied (use --upgrade to upgrade): distribute in ./lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg (from zc.buildout<2dev) > Installing collected packages: zc.buildout > Running setup.py install for zc.buildout > > Skipping installation of /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/lib/python2.7/site-packages/zc/__init__.py (namespace package) > Installing /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/lib/python2.7/site-packages/zc.buildout-1.7.1-py2.7-nspkg.pth > Installing buildout script to /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/bin > Successfully installed zc.buildout > Cleaning up... > > 3. Uninstall it again: > > mauritsvanrees at procyon:foo (git:master?) $ bin/pip uninstall zc.buildout > Uninstalling zc.buildout: > ... > Successfully uninstalled zc.buildout > > 4. Give easy_install another failed chance: > > mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install 'zc.buildout<2dev' > Searching for zc.buildout<2dev > Reading http://pypi.python.org/simple/zc.buildout/ > No local packages or download links found for zc.buildout<2dev > error: Could not find suitable distribution for Requirement.parse('zc.buildout<2dev') > mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install 'zc.buildout==1.7.1' > Searching for zc.buildout==1.7.1 > Reading http://pypi.python.org/simple/zc.buildout/ > error: Connection reset by peer > > 5. easy_install explicitly 1.7.1, now it works: > > mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install zc.buildout==1.7.1 > Searching for zc.buildout==1.7.1 > Reading http://pypi.python.org/simple/zc.buildout/ > Reading http://buildout.org > Best match: zc.buildout 1.7.1 > ... > Processing dependencies for zc.buildout==1.7.1 > Finished processing dependencies for zc.buildout==1.7.1 > > 6. Uninstall it again: > > mauritsvanrees at procyon:foo (git:master?) $ bin/pip uninstall zc.buildout > Uninstalling zc.buildout: > /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/bin/buildout > /Users/mauritsvanrees/community/Products.Poi/src/collective.watcherlist/foo/lib/python2.7/site-packages/zc.buildout-1.7.1-py2.7.egg > Proceed (y/n)? y > Successfully uninstalled zc.buildout > > 6. easy_install the same 1.7.1 again, now it fails: > > mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install zc.buildout==1.7.1 > Searching for zc.buildout==1.7.1 > Reading http://pypi.python.org/simple/zc.buildout/ > No local packages or download links found for zc.buildout==1.7.1 > error: Could not find suitable distribution for Requirement.parse('zc.buildout==1.7.1') > > So: looks random to me. > > pip seems to do it good consistently, but that may be luck. > > > Bootstrapping a buildout also fails, which is why I investigated: > Since My last email or prior to that? I'm in the car on the way to my daughters surgery at CHOP so I don't know how much I'll be able to look today. > $ curl -O http://downloads.buildout.org/1/bootstrap.py > $ python2.7 bootstrap.py > No local packages or download links found for zc.buildout<2dev > error: Could not find suitable distribution for Requirement.parse('zc.buildout<2dev') > An error occurred when trying to install zc.buildout. Look above this message for any errors that were output by easy_install. > > Same with latest version: > > $ curl -O http://downloads.buildout.org/2/bootstrap.py > $ python2.7 bootstrap.py > No local packages or download links found for zc.buildout > error: Could not find suitable distribution for Requirement.parse('zc.buildout') > Traceback (most recent call last): > File "bootstrap.py", line 148, in > repr(cmd)[1:-1]) > Exception: ('Failed to execute command:\n%s', "'/Users/mauritsvanrees/py/bin/python2.7', '-c', 'from setuptools.command.easy_install import main; main()', '-mZqNxd', '/var/folders/b3/4fkrg4_17ms0pwmwh0xy41_w0000gn/T/tmpSgaQlF', 'zc.buildout'") > > If you do 'python2.7 bootstrap.py -h' you will see some ways around this by pointing to a different download location. > > -- > Maurits van Rees: http://maurits.vanrees.org/ > Zest Software: http://zestsoftware.nl > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From m.van.rees at zestsoftware.nl Wed May 29 12:39:55 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 29 May 2013 12:39:55 +0200 Subject: [Distutils] PyPI CDN issues In-Reply-To: <7048D244-0AD3-44C1-BA7D-E00B45E09B64@stufft.io> References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> <7048D244-0AD3-44C1-BA7D-E00B45E09B64@stufft.io> Message-ID: Op 29-05-13 12:34, Donald Stufft schreef: > On May 29, 2013, at 6:18 AM, Maurits van Rees wrote: >> 6. easy_install the same 1.7.1 again, now it fails: >> >> mauritsvanrees at procyon:foo (git:master?) $ bin/easy_install zc.buildout==1.7.1 >> Searching for zc.buildout==1.7.1 >> Reading http://pypi.python.org/simple/zc.buildout/ >> No local packages or download links found for zc.buildout==1.7.1 >> error: Could not find suitable distribution for Requirement.parse('zc.buildout==1.7.1') >> >> So: looks random to me. >> >> pip seems to do it good consistently, but that may be luck. >> >> >> Bootstrapping a buildout also fails, which is why I investigated: >> > > Since My last email or prior to that? This is since your last email. Not sure if it happened prior to that too. > I'm in the car on the way to my daughters surgery at CHOP so I don't know how much I'll be able to look today. That's way more important. Best wishes for that! Thanks, -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From wes at serverdensity.com Wed May 29 13:25:40 2013 From: wes at serverdensity.com (Wes Mason) Date: Wed, 29 May 2013 12:25:40 +0100 Subject: [Distutils] Buildout (still) can't talk to PyPI Message-ID: <585F957A529747CAAC50C505E4CAC04A@boxedice.com> We're getting a similar issue to: http://mail.python.org/pipermail/distutils-sig/2013-May/020927.html For every new build right now, I've recorded the output from "bin/buildout" (initially bootstrap.py would also fail until I added buildout.org to allowed-hosts, as it couldn't get the egg zc.buildout==1.7.1 from pypi), and also the output of trying easy_install inside a virtualenv which suffers the same fate: https://gist.github.com/1stvamp/5669559 -- Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed May 29 13:59:00 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 07:59:00 -0400 Subject: [Distutils] PyPI CDN + Buildout Message-ID: <1369828740.8995.140661237093161.4336B4F0@webmail.messagingengine.com> There's been some issues with buildout and the new PyPI CDN. I'm going to see if I can sort this out. If anyone has buildout successfully working and sees this I'd love to know. To see if this is widespread or specific to a certain set of circumstances. -- Donald Stufft donald at stufft.io From m.van.rees at zestsoftware.nl Wed May 29 14:00:22 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 29 May 2013 14:00:22 +0200 Subject: [Distutils] PyPI CDN issues In-Reply-To: References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> Message-ID: Op 29-05-13 12:18, Maurits van Rees schreef: > $ curl -O http://downloads.buildout.org/2/bootstrap.py > $ python2.7 bootstrap.py > No local packages or download links found for zc.buildout > error: Could not find suitable distribution for > Requirement.parse('zc.buildout') > Traceback (most recent call last): > File "bootstrap.py", line 148, in > repr(cmd)[1:-1]) > Exception: ('Failed to execute command:\n%s', > "'/Users/mauritsvanrees/py/bin/python2.7', '-c', 'from > setuptools.command.easy_install import main; main()', '-mZqNxd', > '/var/folders/b3/4fkrg4_17ms0pwmwh0xy41_w0000gn/T/tmpSgaQlF', > 'zc.buildout'") > > If you do 'python2.7 bootstrap.py -h' you will see some ways around this > by pointing to a different download location. For clarity, this workaround works for me to bootstrap a buildout: python2.7 bootstrap.py --find-links=http://downloads.buildout.org/ -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From m.van.rees at zestsoftware.nl Wed May 29 14:08:18 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 29 May 2013 14:08:18 +0200 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> Message-ID: Op 29-05-13 11:43, Donald Stufft schreef: > > On May 29, 2013, at 5:39 AM, Donald Stufft wrote: >> The cache timeout is an hour unless modifications are made to the >> package which triggers a cache invalidation. >> >> I bet there is a missing Vary for the encoding. I'll check and get >> back to you. >> > There was indeed no Vary headers, I've turned them on and purged the > cache. Please let me know if this behavior doesn't go away. It still works on and off for me. $ curl http://pypi.python.org/simple/pep8/ Links for pep8

Links for pep8

pep8-0.4.2.tar.gz
... Try the same url again: $ curl http://pypi.python.org/simple/pep8/ ??Q??W[??H ... So sometimes it works fine and sometimes it is gzipped . Indeed: $ for i in $(seq 5); do curl -o $i http://pypi.python.org/simple/pep8/; sleep 2; done ... $ ls -l $(seq 5) -rw-r--r-- 1 mauritsvanrees staff 829 May 29 14:05 1 -rw-r--r-- 1 mauritsvanrees staff 3232 May 29 14:05 2 -rw-r--r-- 1 mauritsvanrees staff 3232 May 29 14:05 3 -rw-r--r-- 1 mauritsvanrees staff 829 May 29 14:05 4 -rw-r--r-- 1 mauritsvanrees staff 3232 May 29 14:05 5 -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From donald at stufft.io Wed May 29 14:10:18 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 08:10:18 -0400 Subject: [Distutils] PyPI CDN issues In-Reply-To: References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> Message-ID: <1369829418.10896.140661237097637.0E8EE996@webmail.messagingengine.com> Are you able to consistently reproduce this? Can you get me output with the -v flag to easy_install? E.g. easy_install -v 'zc.buildout<2dev'? Here's what I get: https://gist.github.com/dstufft/5669826 -- Donald Stufft donald at stufft.io From donald at stufft.io Wed May 29 14:16:22 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 08:16:22 -0400 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> Message-ID: <1369829782.12253.140661237100037.61A280E0@webmail.messagingengine.com> Can you get me outputs of curl with the -i flag of it both gzipped and not gzipped? I can't seem to reproduce it. -- Donald Stufft donald at stufft.io From m.van.rees at zestsoftware.nl Wed May 29 14:17:19 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 29 May 2013 14:17:19 +0200 Subject: [Distutils] PyPI CDN issues In-Reply-To: <1369829418.10896.140661237097637.0E8EE996@webmail.messagingengine.com> References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> <1369829418.10896.140661237097637.0E8EE996@webmail.messagingengine.com> Message-ID: Op 29-05-13 14:10, Donald Stufft schreef: > Are you able to consistently reproduce this? Can you get me output with > the -v flag to easy_install? E.g. easy_install -v 'zc.buildout<2dev'? > > Here's what I get: https://gist.github.com/dstufft/5669826 For zc.buildout it now works, but that may be luck. If I try it with a different package it sometimes works and sometimes not, presumably because sometimes a gzipped version of the html file is returned. $ bin/easy_install -v pep8 Searching for pep8 Reading http://pypi.python.org/simple/pep8/ No local packages or download links found for pep8 error: Could not find suitable distribution for Requirement.parse('pep8') Trying the same again: $ bin/easy_install -v pep8 Searching for pep8 Reading http://pypi.python.org/simple/pep8/ Found link: http://pypi.python.org/packages/source/p/pep8/pep8-0.4.2.tar.gz#md5=da8562ef0b18ae9ced210cff742a2d1b ... Best match: pep8 1.4.5 ... And there it goes on to successfully install pep8. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From m.van.rees at zestsoftware.nl Wed May 29 14:30:55 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 29 May 2013 14:30:55 +0200 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: <1369829782.12253.140661237100037.61A280E0@webmail.messagingengine.com> References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> <1369829782.12253.140661237100037.61A280E0@webmail.messagingengine.com> Message-ID: Op 29-05-13 14:16, Donald Stufft schreef: > Can you get me outputs of curl with the -i flag of it both gzipped and > not gzipped? I can't seem to reproduce it. When it works I get this: $ curl -i http://pypi.python.org/simple/pep8/ HTTP/1.1 200 OK Server: nginx/1.1.19 Content-Type: text/html; charset=utf-8 Strict-Transport-Security: max-age=31536000 Cache-Control: max-age=3600, public Via: 1.1 varnish Content-Length: 3232 Accept-Ranges: bytes Date: Wed, 29 May 2013 12:26:01 GMT Via: 1.1 varnish Age: 3594 Connection: keep-alive X-Served-By: cache-s29-SJC2, cache-a39-AMS X-Cache: HIT, HIT X-Cache-Hits: 1, 27 X-Timer: S1369826885.055231571,VS0,VS76,VE76,VE3476514 Vary: Accept-Encoding When I get a gzipped version I get this: $ curl -i http://pypi.python.org/simple/pep8/ HTTP/1.1 200 OK Server: nginx/1.1.19 Content-Type: text/html; charset=utf-8 Content-encoding: gzip Strict-Transport-Security: max-age=31536000 Cache-Control: max-age=3600, public Via: 1.1 varnish Content-Length: 829 Accept-Ranges: bytes Date: Wed, 29 May 2013 12:26:04 GMT Via: 1.1 varnish Age: 2842 Connection: keep-alive X-Served-By: cache-s29-SJC2, cache-a16-AMS X-Cache: MISS, HIT X-Cache-Hits: 0, 23 X-Timer: S1369827522.488703728,VS0,VS77,VE241,VE2841833 Or in a version where you can more easily see the differences: https://gist.github.com/mauritsvanrees/5669913/revisions -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From donald at stufft.io Wed May 29 14:35:41 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 08:35:41 -0400 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> <1369829782.12253.140661237100037.61A280E0@webmail.messagingengine.com> Message-ID: <1369830941.15939.140661237107297.6DAD62BC@webmail.messagingengine.com> I'm in #buildout working on this. We might have sorted out what's going on. It appears we still aren't geting the Vary header sent if Accept-Encoding: gzip is sent. Only if Accept-Encoding: identity is sent. Trying to find my ssh key (ugh laptop I don't normally use) so I can get that header sent on every reply. From marius at pov.lt Wed May 29 14:38:22 2013 From: marius at pov.lt (Marius Gedminas) Date: Wed, 29 May 2013 15:38:22 +0300 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> <1369829782.12253.140661237100037.61A280E0@webmail.messagingengine.com> Message-ID: <20130529123822.GA10130@fridge.pov.lt> On Wed, May 29, 2013 at 02:30:55PM +0200, Maurits van Rees wrote: > Op 29-05-13 14:16, Donald Stufft schreef: > >Can you get me outputs of curl with the -i flag of it both gzipped and > >not gzipped? I can't seem to reproduce it. > > When it works I get this: > > $ curl -i http://pypi.python.org/simple/pep8/ > HTTP/1.1 200 OK ... > Content-Length: 3232 ... > Vary: Accept-Encoding > > When I get a gzipped version I get this: > > $ curl -i http://pypi.python.org/simple/pep8/ > HTTP/1.1 200 OK ... > Content-encoding: gzip ... There's a discussion about this in #buildout on irc.freenode.net It looks like the upstream server is misconfigured and doesn't emit Vary: Accept-Encoding when a client does a GET with Accept-Encoding: gzip and gets back a gzipped response (which then gets cached and served to everyone for one hour). Donald Stufft is aware of this and is trying to fix it from his phone while sitting in an OR waiting room. I'm in awe. Marius Gedminas -- IBM motto: "TEN vowels? Don't you know vowels are scrd?" -- Linus Torvalds -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From donald at stufft.io Wed May 29 15:00:48 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 09:00:48 -0400 Subject: [Distutils] PyPI CDN + Buildout In-Reply-To: <1369828740.8995.140661237093161.4336B4F0@webmail.messagingengine.com> References: <1369828740.8995.140661237093161.4336B4F0@webmail.messagingengine.com> Message-ID: <1369832448.20653.140661237117713.2756F7B5@webmail.messagingengine.com> I believe this should be fixed now. I ensured Vary: Accept-Encoding was called even when responses were gzipped. I also purged the cache so bad entries should be gone. IF you have any more issues after this email please tell me. -- Donald Stufft donald at stufft.io From donald at stufft.io Wed May 29 15:01:01 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 09:01:01 -0400 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: <20130529123822.GA10130@fridge.pov.lt> References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> <1369829782.12253.140661237100037.61A280E0@webmail.messagingengine.com> <20130529123822.GA10130@fridge.pov.lt> Message-ID: <1369832461.20698.140661237118117.585F711B@webmail.messagingengine.com> I believe this should be fixed now. I ensured Vary: Accept-Encoding was called even when responses were gzipped. I also purged the cache so bad entries should be gone. IF you have any more issues after this email please tell me. -- Donald Stufft donald at stufft.io On Wed, May 29, 2013, at 08:38 AM, Marius Gedminas wrote: > On Wed, May 29, 2013 at 02:30:55PM +0200, Maurits van Rees wrote: > > Op 29-05-13 14:16, Donald Stufft schreef: > > >Can you get me outputs of curl with the -i flag of it both gzipped and > > >not gzipped? I can't seem to reproduce it. > > > > When it works I get this: > > > > $ curl -i http://pypi.python.org/simple/pep8/ > > HTTP/1.1 200 OK > ... > > Content-Length: 3232 > ... > > Vary: Accept-Encoding > > > > When I get a gzipped version I get this: > > > > $ curl -i http://pypi.python.org/simple/pep8/ > > HTTP/1.1 200 OK > ... > > Content-encoding: gzip > ... > > There's a discussion about this in #buildout on irc.freenode.net > > It looks like the upstream server is misconfigured and doesn't emit > > Vary: Accept-Encoding > > when a client does a GET with Accept-Encoding: gzip and gets back a > gzipped response (which then gets cached and served to everyone for one > hour). > > Donald Stufft is aware of this and is trying to fix it from his phone > while sitting in an OR waiting room. > > I'm in awe. > > Marius Gedminas > -- > IBM motto: "TEN vowels? Don't you know vowels are scrd?" > -- Linus Torvalds > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature) From donald at stufft.io Wed May 29 15:01:21 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 09:01:21 -0400 Subject: [Distutils] PyPI CDN issues In-Reply-To: References: <51A5BAC5.6090600@zopyx.com> <2F11BB5C-EF91-4F2C-B23B-AA08603AC909@stufft.io> <1369829418.10896.140661237097637.0E8EE996@webmail.messagingengine.com> Message-ID: <1369832481.20659.140661237118377.2ABF58C0@webmail.messagingengine.com> I believe this should be fixed now. I ensured Vary: Accept-Encoding was called even when responses were gzipped. I also purged the cache so bad entries should be gone. IF you have any more issues after this email please tell me. -- Donald Stufft donald at stufft.io On Wed, May 29, 2013, at 08:17 AM, Maurits van Rees wrote: > Op 29-05-13 14:10, Donald Stufft schreef: > > Are you able to consistently reproduce this? Can you get me output with > > the -v flag to easy_install? E.g. easy_install -v 'zc.buildout<2dev'? > > > > Here's what I get: https://gist.github.com/dstufft/5669826 > > For zc.buildout it now works, but that may be luck. > > If I try it with a different package it sometimes works and sometimes > not, presumably because sometimes a gzipped version of the html file is > returned. > > $ bin/easy_install -v pep8 > Searching for pep8 > Reading http://pypi.python.org/simple/pep8/ > No local packages or download links found for pep8 > error: Could not find suitable distribution for Requirement.parse('pep8') > > Trying the same again: > > $ bin/easy_install -v pep8 > Searching for pep8 > Reading http://pypi.python.org/simple/pep8/ > Found link: > http://pypi.python.org/packages/source/p/pep8/pep8-0.4.2.tar.gz#md5=da8562ef0b18ae9ced210cff742a2d1b > ... > Best match: pep8 1.4.5 > ... > > And there it goes on to successfully install pep8. > > > -- > Maurits van Rees: http://maurits.vanrees.org/ > Zest Software: http://zestsoftware.nl > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From m.van.rees at zestsoftware.nl Wed May 29 15:16:49 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 29 May 2013 15:16:49 +0200 Subject: [Distutils] Buildout can't talk to PyPI any more? In-Reply-To: <1369832461.20698.140661237118117.585F711B@webmail.messagingengine.com> References: <20130529081943.GB30835@fridge.pov.lt> <20130529091823.GA7606@fridge.pov.lt> <338251BD-292E-4C13-8B82-FF8232179994@stufft.io> <09141395-81C1-4C0E-8B91-164F3AF1A3A7@stufft.io> <1369829782.12253.140661237100037.61A280E0@webmail.messagingengine.com> <20130529123822.GA10130@fridge.pov.lt> <1369832461.20698.140661237118117.585F711B@webmail.messagingengine.com> Message-ID: Op 29-05-13 15:01, Donald Stufft schreef: > I believe this should be fixed now. I ensured Vary: Accept-Encoding was > called even when responses were gzipped. > > I also purged the cache so bad entries should be gone. IF you have any > more issues after this email please tell me. Looks fine here now. I downloaded several packages ten times with curl and it all went fine. Well, I got a timeout trying to get distribute, but that was only one out of ten tries. Besides, the CDN should help there anyway. Thanks a lot! -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From donald at stufft.io Wed May 29 15:28:43 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 09:28:43 -0400 Subject: [Distutils] Small Usability/UX Change to PyPI Message-ID: <1369834123.25639.140661237131513.63E815ED@webmail.messagingengine.com> Just deployed a small update to PyPI that sets autofocus on the search form. This'll mean that you don't need to manually move to the search form upon page load and should be able to start typing immediately. -- Donald Stufft donald at stufft.io From dholth at gmail.com Wed May 29 16:14:33 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 May 2013 10:14:33 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: On Tue, May 28, 2013 at 11:12 PM, Nick Coghlan wrote: > A couple of significant upcoming changes: > > "build label" will be renamed as "source label" (since it refers to > the common unbuilt source rather than a specific build) > "version URL" will be renamed as "source URL" (same rationale) > > The current names are ambiguous as to whether they refer to the source > code for the version or can be used to refer to built versions. Since > they're specifically for source references (you need to add at least > PEP 425 compatibility tags to construct a built reference), it makes > sense to change the names. > > A more minor change is that the "organization" type/role for contacts > will go away. Organization will be able to have any of the defined > roles (author, maintainer, contributor) and if we later decide we need > a programmatic means to distinguish abstract organisations from flesh > and blood humans we can consider adding a new mechanism. > > Cheers, > Nick. " Request the test extra to also install test_requires test_may_require " If test requirements are not actually extras then I would prefer having no special-cased extra names at all, or a special extra-like syntax used for pip-install'ing different categories of dependencies. Is there a function to get the metadata keys from the category name? It would be something like: f('test') -> 'test_requires', 'test_may_require' def f(category): if category == 'install': category = '' conditional = [category, 'may_require'] unconditional = [category, '_requires'] return ('_'.join(unconditional).strip('_'), '_'.join(conditional).strip('_')) From ncoghlan at gmail.com Wed May 29 16:18:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 May 2013 00:18:46 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <06FE1104-014B-4EA1-9CAF-6BEC0FAFD4F1@mac.com> References: <06FE1104-014B-4EA1-9CAF-6BEC0FAFD4F1@mac.com> Message-ID: On Tue, May 28, 2013 at 12:44 AM, Ronald Oussoren wrote: > > On 27 May, 2013, at 13:36, Nick Coghlan wrote: > >> After preliminary reviews by Donald and Daniel, I have now pushed the >> first complete draft of the JSON-based metadata 2.0 proposal to >> python.org >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > Could platform_release be added to the set of environment markers? On OSX platform.release() returns a kernel version number, while platform.version() returns a string that cannot easily be used as a marker. Heh, I just checked Linux and it's the same - I don't know where the original list came from, but adding platform_release sounds good to me. > It might also be nice to have markers for the marketing name and version of the OS next to the kernel name and release (for example Solaris 8 vs. SunOS 5.8, or darwin 12.3.0 vs. OSX 10.8.3), but that would probably need a PEP of its own to add the functionality to the stdlib before it is used in metadata. > > BTW. Why can marker expressions use only a limited subset of operators? In particular <, <=, > and >= are not allowed and would be nice to have to specify packages that need a minimal platform version (parts of PyObjC have a dependency on the OSX release because the wrapped library is available in a subset of releases). Listing all OS releases would be fairly useless, platform.release() is too specific to use with '==' as new patch releases of OSX can introduce new micro versions of the kernel. Simple string comparison also works pretty well with Python versions, too. I think it makes sense to allow them. >> PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ > > The versioning spec mentions that distribution tools may refuse to publish distributions that pin the versions of dependencies. I understand why this is needed, and agree in general, but have a usecase that I don't know how to express without pinning. > > In particular, PyObjC consists of a number of distributions (pyobjc-core, pyobjc-framework-Cocoa, ...) and an umbrella package (pyobjc) what depends on the various distributions to make it easier to install all of PyObjC. The umbrella package currently pins the versions of subpackages to ensure that "pip install pyobjc==2.5.1" installs exactly that version of the entire project. When I'd use the "compatible release" specifier I can no longer easily ensure that users can install an exact version of the entire project, other than by hacking the system: specify a compatible version with an additional level that isn't used by the project (for example ~=2.5.2.0). What is the correct way to create an umberella project without getting yelled at by distribution tools? As noted in a previous email, I've added a specific field for this. It also highlighted some issues with the way version pinning was documented in general, so it led to some other changes (like renaming build labels and build references to source labels and source references). The latest drafts should be up on python.org shortly (I just pushed them to both the PEPs repo and bitbucket) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed May 29 16:25:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 May 2013 00:25:07 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: On Thu, May 30, 2013 at 12:14 AM, Daniel Holth wrote: > Request the test extra to also install > > test_requires > test_may_require > " > > If test requirements are not actually extras then I would prefer > having no special-cased extra names at all, or a special extra-like > syntax used for pip-install'ing different categories of dependencies. No, I still like the idea of using the existing extras syntax to request their installation when you do want them, rather than needing a special mechanism in the command line tools. The declaration just got broken out to a separate field as otherwise it wasn't clear whether test dependencies should be listed in may_require, dev_requires or dev_may_require. > Is there a function to get the metadata keys from the category name? > It would be something like: > > f('test') -> 'test_requires', 'test_may_require' > > def f(category): > > if category == 'install': category = '' > conditional = [category, 'may_require'] > unconditional = [category, '_requires'] > return ('_'.join(unconditional).strip('_'), '_'.join(conditional).strip('_')) Possibly, but API details aren't a question for this PEP. If you look at the deployment scenarios in the PEP, that's not actually the way the mapping works anyway (development mode gets everything, not just the dev dependencies). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Wed May 29 16:32:52 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 May 2013 10:32:52 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: On Wed, May 29, 2013 at 10:25 AM, Nick Coghlan wrote: > On Thu, May 30, 2013 at 12:14 AM, Daniel Holth wrote: >> Request the test extra to also install >> >> test_requires >> test_may_require >> " >> >> If test requirements are not actually extras then I would prefer >> having no special-cased extra names at all, or a special extra-like >> syntax used for pip-install'ing different categories of dependencies. > > No, I still like the idea of using the existing extras syntax to > request their installation when you do want them, rather than needing > a special mechanism in the command line tools. > > The declaration just got broken out to a separate field as otherwise > it wasn't clear whether test dependencies should be listed in > may_require, dev_requires or dev_may_require. I just mean something like pip install package[:test:], using any character that is not allowed in extra names to avoid a collision. >> Is there a function to get the metadata keys from the category name? >> It would be something like: >> >> f('test') -> 'test_requires', 'test_may_require' >> >> def f(category): >> >> if category == 'install': category = '' >> conditional = [category, 'may_require'] >> unconditional = [category, '_requires'] >> return ('_'.join(unconditional).strip('_'), '_'.join(conditional).strip('_')) > > Possibly, but API details aren't a question for this PEP. If you look > at the deployment scenarios in the PEP, that's not actually the way > the mapping works anyway (development mode gets everything, not just > the dev dependencies). I think the naming of the two keys is slightly inelegant, with the 'may' in the middle and the requires / require, but there aren't any short english words that would look good simply appended to the end 'test_requires', 'test_requires_conditional'. From donald at stufft.io Wed May 29 16:39:53 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 10:39:53 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: <1369838393.6446.140661237162361.236F7EE6@webmail.messagingengine.com> I support using :test, or :test: or whatever to mark it as a "special" name. -- Donald Stufft donald at stufft.io From ncoghlan at gmail.com Wed May 29 16:50:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 May 2013 00:50:53 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: On Thu, May 30, 2013 at 12:32 AM, Daniel Holth wrote: > On Wed, May 29, 2013 at 10:25 AM, Nick Coghlan wrote: >> On Thu, May 30, 2013 at 12:14 AM, Daniel Holth wrote: >>> Request the test extra to also install >>> >>> test_requires >>> test_may_require >>> " >>> >>> If test requirements are not actually extras then I would prefer >>> having no special-cased extra names at all, or a special extra-like >>> syntax used for pip-install'ing different categories of dependencies. >> >> No, I still like the idea of using the existing extras syntax to >> request their installation when you do want them, rather than needing >> a special mechanism in the command line tools. >> >> The declaration just got broken out to a separate field as otherwise >> it wasn't clear whether test dependencies should be listed in >> may_require, dev_requires or dev_may_require. > > I just mean something like pip install package[:test:], using any > character that is not allowed in extra names to avoid a collision. Hmm, I'll have to think about that one. It doesn't cause any data modelling issues any more (since the dedicated fields have been broken out to store the test dependencies), and you're right, lifting the "any name except test" restriction on the extras field itself would be nice. >>> Is there a function to get the metadata keys from the category name? >>> It would be something like: >>> >>> f('test') -> 'test_requires', 'test_may_require' >>> >>> def f(category): >>> >>> if category == 'install': category = '' >>> conditional = [category, 'may_require'] >>> unconditional = [category, '_requires'] >>> return ('_'.join(unconditional).strip('_'), '_'.join(conditional).strip('_')) >> >> Possibly, but API details aren't a question for this PEP. If you look >> at the deployment scenarios in the PEP, that's not actually the way >> the mapping works anyway (development mode gets everything, not just >> the dev dependencies). > > I think the naming of the two keys is slightly inelegant, with the > 'may' in the middle and the requires / require, but there aren't any > short english words that would look good simply appended to the end > 'test_requires', 'test_requires_conditional'. Yeah, I picked the syntax I did for readability. I've seen similar APIs along the lines of "requires" and "requires_maybe" and they're ugly. In terms of how you deal with it programmatically, you would likely want to have your mandatory and conditional handling separated out anyway, so iterators are a natural fit: def iter_dependencies(metadata, field, prefixes): for prefix in prefixes: if prefix: field_name = prefix + "_" + field else: field_name = field data = metadata.get(field_name) if data is None: continue yield from data def iter_requires(metadata, prefixes): yield from iter_dependencies(metadata, "requires", prefixes) def iter_may_require(metadata, prefixes): yield from iter_dependencies(metadata, "may_require", prefixes) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed May 29 17:23:00 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 11:23:00 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: <1369840980.15142.140661237180577.6B125765@webmail.messagingengine.com> I feel like we are still conflating some names and use cases here with source_label, source_url, source reference, and the is operator. source_label // source_url - References to the source that is associated with this release. Must not point to anything besides source (no whls, no other built or platform specific artifact). is // source reference - This is where my real issue comes into play. I almost think that it doesn't make any sense to allow an is comparison to a source label. In Python `is` is an identity function and a source label is not a directly installable thing. It must be built first (even if building involves simply copying .py files into a whl). I don't like using the word "source" here because when pointing at an url it could be a whl file, it could be an egg, it could be a source archive. I'm not sure what should be done about the non url form of ``is``. The url form makes sense to me because an url is (theortically) an identity pointer to an exact installable. However a source_label is not, it could have any number of installable files associated with it and the one you install may be dependent upon what OS you're using or what Python version your using. -- Donald Stufft donald at stufft.io From eric at trueblade.com Wed May 29 17:19:04 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 29 May 2013 11:19:04 -0400 Subject: [Distutils] Small Usability/UX Change to PyPI In-Reply-To: <1369834123.25639.140661237131513.63E815ED@webmail.messagingengine.com> References: <1369834123.25639.140661237131513.63E815ED@webmail.messagingengine.com> Message-ID: <51A61C68.6060301@trueblade.com> On 05/29/2013 09:28 AM, Donald Stufft wrote: > Just deployed a small update to PyPI that sets autofocus on the search > form. This'll mean that you don't need to manually move to the search > form upon page load and should be able to start typing immediately. Thank you, thank you, thank you! Eric. From chris at simplistix.co.uk Wed May 29 17:40:18 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 29 May 2013 16:40:18 +0100 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server In-Reply-To: <87ip227k9y.fsf@myhost.lan> References: <87ip227k9y.fsf@myhost.lan> Message-ID: <51A62162.1000901@simplistix.co.uk> On 29/05/2013 00:32, Ralf Schmitt wrote: > I've just uploaded pypiserver 1.1.1 to the python package index. > > pypiserver is a minimal PyPI compatible server. It can be used to serve > a set of packages and eggs to easy_install or pip. > > pypiserver is easy to install (i.e. just 'pip install pypiserver'). It > doesn't have any external dependencies. > > https://pypi.python.org/pypi/pypiserver/ should contain enough > information to easily get you started running your own PyPI server in a > Looks very interesting! How does it compare to http://pypi.python.org/pypi/devpi-server? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed May 29 17:48:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 May 2013 01:48:00 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <1369840980.15142.140661237180577.6B125765@webmail.messagingengine.com> References: <1369840980.15142.140661237180577.6B125765@webmail.messagingengine.com> Message-ID: On 30 May 2013 01:23, "Donald Stufft" wrote: > > I feel like we are still conflating some names and use cases here with > source_label, source_url, source reference, and the is operator. > > source_label // source_url - References to the source that is associated > with this release. Must not point to anything besides source (no whls, > no other built or platform specific artifact). > > is // source reference - This is where my real issue comes into play. > > I almost think that it doesn't make any sense to allow an is comparison > to a source label. In Python `is` is an identity function and a source > label is not a directly installable thing. It must be built first (even > if building involves simply copying .py files into a whl). > > I don't like using the word "source" here because when pointing at an > url it could be a whl file, it could be an egg, it could be a source > archive. > > I'm not sure what should be done about the non url form of ``is``. The > url form makes sense to me because an url is (theortically) an identity > pointer to an exact installable. However a source_label is not, it could > have any number of installable files associated with it and the one you > install may be dependent upon what OS you're using or what Python > version your using. The whole of the metadata operates at the source identification level - it's up to the installation tools to work out how best to satisfy that given the installation context. I only allowed build URLs at all because it seemed silly to force installation tools to come up with a distinct format for them. However, if allowing both appears to be too confusing, that's the half that will go away, leaving it up to the integration tools to say how to nominate particular binaries. Source references themselves are in danger of going away entirely as being not worth the hassle. Cheers, Nick. > > -- > Donald Stufft > donald at stufft.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 29 17:52:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 May 2013 01:52:06 +1000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: <1369840980.15142.140661237180577.6B125765@webmail.messagingengine.com> Message-ID: On 30 May 2013 01:47, "Daniel Holth" wrote: > > In 440, < 2.0 means < 2.0, != 2.0.* to avoid dev releases. Would it be > equivalent for < 2.0 to expand to the smallest possible release in the > 2.0 series 2.0.dev0? Yes, but that approach doesn't work to exclude post and maintenance releases in the > 2.0 case. The PEP uses the expansion it does so that the greater than and less operations use similar definitions. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed May 29 17:47:41 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 May 2013 11:47:41 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <1369840980.15142.140661237180577.6B125765@webmail.messagingengine.com> References: <1369840980.15142.140661237180577.6B125765@webmail.messagingengine.com> Message-ID: In 440, < 2.0 means < 2.0, != 2.0.* to avoid dev releases. Would it be equivalent for < 2.0 to expand to the smallest possible release in the 2.0 series 2.0.dev0? From donald at stufft.io Wed May 29 18:08:32 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 May 2013 12:08:32 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: <1369840980.15142.140661237180577.6B125765@webmail.messagingengine.com> Message-ID: <1369843712.23957.140661237202417.34BCA18C@webmail.messagingengine.com> On Wed, May 29, 2013, at 11:48 AM, Nick Coghlan wrote: I only allowed build URLs at all because it seemed silly to force installation tools to come up with a distinct format for them. However, if allowing both appears to be too confusing, that's the half that will go away, leaving it up to the integration tools to say how to nominate particular binaries. Source references themselves are in danger of going away entirely as being not worth the hassle. I understand it's supposed to operate at the release level (vs the file level) but I think it needs to operate at both. The common case is going to be "I'm referencing the thing I want to install by name and version". However I think it should be a supported operation to say "I want to install exactly this file. Nothing else will do". ``foo (is [1]https://example.com/foo-1.0.tar.gz)`` solves that. However that doesn't jive with ``foo (is )`` would provide that. I don't see much value in being able to install by referencing the source_label (to me it's more of an identifier to allow tools like RPM to trace a PyPI release back to it's origin). I do however see immense value in being able to single out a particular file you want to install either via url or a uniquely identifier (for project) build tag/label. References 1. https://example.com/foo-1.0.tar.gz -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed May 29 20:34:37 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 29 May 2013 14:34:37 -0400 Subject: [Distutils] Small Usability/UX Change to PyPI In-Reply-To: <1369834123.25639.140661237131513.63E815ED@webmail.messagingengine.com> References: <1369834123.25639.140661237131513.63E815ED@webmail.messagingengine.com> Message-ID: On Wed, May 29, 2013 at 9:28 AM, Donald Stufft wrote: > Just deployed a small update to PyPI that sets autofocus on the search > form. This'll mean that you don't need to manually move to the search > form upon page load and should be able to start typing immediately. That is so trivial yet so awesome. Thank you! From holger at merlinux.eu Wed May 29 20:36:33 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 29 May 2013 18:36:33 +0000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: <20130529183633.GM7789@merlinux.eu> Hi Nick, On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: > After preliminary reviews by Donald and Daniel, I have now pushed the > first complete draft of the JSON-based metadata 2.0 proposal to > python.org > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ After a first quick read i am wondering if i missed something with respect to test tools. There are some fields which specify dependencies required for running tests, but there is nothing that would tell which test runner to use, which test command to invoke, or am i missing something? Basically i am wondering how PEP426 could be useful/used by tox. best, holger From dholth at gmail.com Wed May 29 21:15:25 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 May 2013 15:15:25 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <20130529183633.GM7789@merlinux.eu> References: <20130529183633.GM7789@merlinux.eu> Message-ID: On Wed, May 29, 2013 at 2:36 PM, holger krekel wrote: > Hi Nick, > > On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: >> After preliminary reviews by Donald and Daniel, I have now pushed the >> first complete draft of the JSON-based metadata 2.0 proposal to >> python.org >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > After a first quick read i am wondering if i missed something > with respect to test tools. There are some fields which specify > dependencies required for running tests, but there is nothing that > would tell which test runner to use, which test command to invoke, > or am i missing something? > > Basically i am wondering how PEP426 could be useful/used by tox. > best, The first thing we might do is to have setuptools expose its test_suite argument as "extensions" : { "setuptools": { "test_suite": "callable.name" } }. I think we need the next version of the metadata or sdist 2.0 to really do a better job than just running "setup.py test"; right now the tests usually only make sense in the context of an unpacked sdist. Input appreciated. From holger at merlinux.eu Wed May 29 21:33:48 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 29 May 2013 19:33:48 +0000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: <20130529183633.GM7789@merlinux.eu> Message-ID: <20130529193348.GN7789@merlinux.eu> On Wed, May 29, 2013 at 15:15 -0400, Daniel Holth wrote: > On Wed, May 29, 2013 at 2:36 PM, holger krekel wrote: > > Hi Nick, > > > > On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: > >> After preliminary reviews by Donald and Daniel, I have now pushed the > >> first complete draft of the JSON-based metadata 2.0 proposal to > >> python.org > >> > >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > > > After a first quick read i am wondering if i missed something > > with respect to test tools. There are some fields which specify > > dependencies required for running tests, but there is nothing that > > would tell which test runner to use, which test command to invoke, > > or am i missing something? > > > > Basically i am wondering how PEP426 could be useful/used by tox. > > best, > > The first thing we might do is to have setuptools expose its > test_suite argument as "extensions" : { "setuptools": { "test_suite": > "callable.name" } }. The way tox specifies testing is to allow arbitrary test commands not just unittest test suites. People use py.test, make and whatnot to run tests. > I think we need the next version of the metadata or sdist 2.0 to > really do a better job than just running "setup.py test"; right now > the tests usually only make sense in the context of an unpacked sdist. > Input appreciated. We certainly don't want to advocate using "setup.py" for running tests as we want to get way from the neccessity for this file to exist (correct me if i am wrong with this presumption). The metadata should be rich enough to support tools like tox to perform the testing, much like pip for installations. holger From george at reilly.org Wed May 29 21:33:41 2013 From: george at reilly.org (George V. Reilly) Date: Wed, 29 May 2013 12:33:41 -0700 Subject: [Distutils] Using code from a dependency in setup.py Message-ID: I have a little web framework, let's call it Bread, which is used to build applications like Jam, Marmalade, PeanutButter, and other toppings. Bread both *builds* and *serves* these applications. I'm trying to figure out how to make the applications' `setup.py`s work, given the following requirements: * The apps depend upon Bread, via [setuptool's install_requires]( http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies) * To build an application at development time, Bread reads some config and then emits assets (HTML, JS, CSS, images, etc) to the application's `output` directory. In other words, `bread devserver` reads `Jam/bread.yaml` and assembles assets in `Jam/output`, then serves the application (via Flask, but that's not otherwise pertinent). * In order to build a *deployable* Jam application, I want to invoke Bread during `python setup.py install` of Jam, to build `Jam/output`. In production, Jam should not need to build anything. * I've defined a custom `bdist_egg` setup command where `initialize_options` imports Bread, invokes the builder, then sets `self.distribution.data_files` with the appropriate tuples, before calling the base class. (And *that* was no fun to figure out.) * Right now, the `bdist_egg` is defined in Jam's `setup.py`. I want to move this and other boilerplate code into `bread.setup`, so that I can reuse it in Marmalade, PeanutButter, etc. * Potentially, this means that I'm now importing Bread code before Bread has been installed. This will surely arise in a clean install, such as a fresh virtualenv on a build machine. Can this be done with Distutils / setuptools / Distribute? -- /George V. Reilly george at reilly.org Twitter: @georgevreilly http://www.georgevreilly.com/blog http://blogs.cozi.com/tech -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed May 29 22:12:00 2013 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 30 May 2013 08:12:00 +1200 Subject: [Distutils] Using code from a dependency in setup.py In-Reply-To: References: Message-ID: On 30 May 2013 07:33, George V. Reilly wrote: > Can this be done with Distutils / setuptools / Distribute? I believe that setup_requires can do what you want, in principle - AIUI that describes what modules are needed to run setup correctly : of course you have to bootstrap manually - but installers can examine the EGG_INFO once it's created. I suspect only ezsetup does that though - though I'd be delighted to be wrong. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services From dholth at gmail.com Wed May 29 22:26:42 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 May 2013 16:26:42 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <20130529193348.GN7789@merlinux.eu> References: <20130529183633.GM7789@merlinux.eu> <20130529193348.GN7789@merlinux.eu> Message-ID: On Wed, May 29, 2013 at 3:33 PM, holger krekel wrote: > On Wed, May 29, 2013 at 15:15 -0400, Daniel Holth wrote: >> On Wed, May 29, 2013 at 2:36 PM, holger krekel wrote: >> > Hi Nick, >> > >> > On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: >> >> After preliminary reviews by Donald and Daniel, I have now pushed the >> >> first complete draft of the JSON-based metadata 2.0 proposal to >> >> python.org >> >> >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ >> > >> > After a first quick read i am wondering if i missed something >> > with respect to test tools. There are some fields which specify >> > dependencies required for running tests, but there is nothing that >> > would tell which test runner to use, which test command to invoke, >> > or am i missing something? >> > >> > Basically i am wondering how PEP426 could be useful/used by tox. >> > best, >> >> The first thing we might do is to have setuptools expose its >> test_suite argument as "extensions" : { "setuptools": { "test_suite": >> "callable.name" } }. > > The way tox specifies testing is to allow arbitrary test commands not > just unittest test suites. People use py.test, make and whatnot > to run tests. tox could shove a JSON version of its config into the metadata as an extension. If "something wrapped in a unittest suite" isn't the interface for running tests (in the most boring way possible, with better runners for the developers not the users of the package), then what should it be? >> I think we need the next version of the metadata or sdist 2.0 to >> really do a better job than just running "setup.py test"; right now >> the tests usually only make sense in the context of an unpacked sdist. >> Input appreciated. > > We certainly don't want to advocate using "setup.py" for running tests as > we want to get way from the neccessity for this file to exist (correct me > if i am wrong with this presumption). The metadata should be rich > enough to support tools like tox to perform the testing, much > like pip for installations. The first priority is to get rid of "setup.py install" since that is where most of the trouble comes from. It's not that big of a task to relegate setup.py to its proper place as a build script. At the same time it will eventually become possible to use non-distutils-derived build tools without having to emulate a distutils setup.py. Wouldn't we continue to type tox, or nose, or py.test, with the standard hook being mainly for automated tools that just want to verify that something still works before building / installing? From ct at gocept.com Wed May 29 22:44:28 2013 From: ct at gocept.com (Christian Theune) Date: Wed, 29 May 2013 22:44:28 +0200 Subject: [Distutils] changelog / CDN inconsistency References: <92ABFF42-F2A7-4CEF-B12E-7F6C3EF9C818@coderanger.net> <20130527120846.GA7789@merlinux.eu> <3302D9F4-1000-447E-A9C6-E83D2FC07AC4@stufft.io> <20130527185453.GG7789@merlinux.eu> <20130527191839.GI7789@merlinux.eu> <35514077-3F0A-45C4-8DDA-19D9F51ECBFC@coderanger.net> <20130527202023.GJ7789@merlinux.eu> <982FCB5D-F518-4AC3-ACC6-37AECBA4CC6E@coderanger.net> <3E6A3097-FAD0-4DAE-A21E-A259219C0736@gocept.com> <51A4DBC6.4010809@v.loewis.de> Message-ID: On 2013-05-28 16:31:02 +0000, Martin v. L?wis said: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Am 28.05.13 11:04, schrieb Christian Theune: >> So - what's the next step that can happen ASAP? > > In addition to the changes Donald already did, I think it would be > wise to restart mirroring at min(last_serial, last_mirroring - 1 minute). We dropped considering serials as timestamps. One of the reasons pep381client was getting inconsistent was that was abusing something that is a serial as a timestamp. Comparing serials and times does not make (too much) sense any longer. I don't want to introduce this again. > This will cause any simple pages to be re-downloaded that had been > updated just before the last mirror run completed. > > If you aren't checking md5sums of files after download, you should > (I always wanted to put this into pep381client). Right, I managed to include this in bandersnatch. Christian -------------- next part -------------- An HTML attachment was scrubbed... URL: From zooko at zooko.com Wed May 29 23:48:00 2013 From: zooko at zooko.com (zooko) Date: Thu, 30 May 2013 01:48:00 +0400 Subject: [Distutils] Using code from a dependency in setup.py In-Reply-To: References: Message-ID: <20130529214759.GD30470@zooko.com> Here is my explanation of why it can be problematic if you import Bread in Bread's setup.py: https://bugs.launchpad.net/nevow/+bug/812537/comments/3 If you do want to import dependencies of Bread in Bread's setup.py, and if your build tool (e.g. setuptools) supports it, then you could add the dependencies to the "setup_requires" list. If possible, maybe you could move some of this work from setup.py-time (whether build-time or install-time) to "first time I run" or a custom "initialize me now" command of Bread. I think setup.py is a sub-optimal place to do stuff, because there are a lot of complicated things going then and there, and because it is very inconvenient to write unit tests of setup.py behavior. Regards, Zooko From george at reilly.org Thu May 30 02:40:02 2013 From: george at reilly.org (George V. Reilly) Date: Wed, 29 May 2013 17:40:02 -0700 Subject: [Distutils] Using code from a dependency in setup.py In-Reply-To: <20130529214759.GD30470@zooko.com> References: <20130529214759.GD30470@zooko.com> Message-ID: Actually, Bread's setup is fine; it's not doing anything interesting. It's Jam's setup that's the problem. Robert's mention of `setup_requires` led me to http://stackoverflow.com/a/12061891/6364, which gave me the hint I needed: create a separate `Distribution` object before calling `setup` which defines the `setup_requires` entries. Jam's `setup.py` now looks like: from setuptools import setup, dist dist.Distribution(dict(setup_requires='Bread')) from bread.setup_topping import * setup( name='Jam', version='0.2', long_description=open('README.md').read(), **topping_setup_options ) # Remove *.egg left by bootstrapping Bread cleanup_bread_bootstrap() And `bread/setup_topping.py` looks like: from setuptools.command.bdist_egg import bdist_egg as _bdist_egg import os, fnmatch, glob, shutil def recursive_data_files(treeroot, pattern): results = [] for base, dirs, files in os.walk(treeroot): goodfiles = fnmatch.filter(files, pattern) if goodfiles: results.append((base, [os.path.join(base, f) for f in goodfiles])) return results def make_data_files(output='output'): return ( [('', ['bread.yaml'])] + recursive_data_files(output, '*') ) class bdist_egg(_bdist_egg): def initialize_options(self): bake_bread() # build files to './output' self.distribution.data_files = make_data_files() _bdist_egg.initialize_options(self) topping_setup_options = dict( cmdclass={ 'bdist_egg': bdist_egg, }, install_requires=[ 'Bread', ], zip_safe=False, ) def cleanup_bread_bootstrap(root='.'): for f in glob.glob(os.path.join(os.path.abspath(root), '*.egg')): if os.path.isdir(f): shutil.rmtree(f) # Egg directory else: os.remove(f) # Zipped Egg On Wed, May 29, 2013 at 2:48 PM, zooko wrote: > Here is my explanation of why it can be problematic if you import Bread in > Bread's setup.py: > > https://bugs.launchpad.net/nevow/+bug/812537/comments/3 > > If you do want to import dependencies of Bread in Bread's setup.py, and if > your > build tool (e.g. setuptools) supports it, then you could add the > dependencies > to the "setup_requires" list. > > If possible, maybe you could move some of this work from setup.py-time > (whether > build-time or install-time) to "first time I run" or a custom "initialize > me > now" command of Bread. I think setup.py is a sub-optimal place to do stuff, > because there are a lot of complicated things going then and there, and > because > it is very inconvenient to write unit tests of setup.py behavior. > > Regards, > > Zooko > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Thu May 30 10:19:36 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 30 May 2013 08:19:36 +0000 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server In-Reply-To: <51A62162.1000901@simplistix.co.uk> References: <87ip227k9y.fsf@myhost.lan> <51A62162.1000901@simplistix.co.uk> Message-ID: <20130530081936.GP7789@merlinux.eu> On Wed, May 29, 2013 at 16:40 +0100, Chris Withers wrote: > On 29/05/2013 00:32, Ralf Schmitt wrote: > >I've just uploaded pypiserver 1.1.1 to the python package index. > > > >pypiserver is a minimal PyPI compatible server. It can be used to serve > >a set of packages and eggs to easy_install or pip. > > > >pypiserver is easy to install (i.e. just 'pip install pypiserver'). It > >doesn't have any external dependencies. > > > >https://pypi.python.org/pypi/pypiserver/ should contain enough > >information to easily get you started running your own PyPI server in a > > > > Looks very interesting! > > How does it compare to http://pypi.python.org/pypi/devpi-server? the main differences as i see them (note i am the devpi-server author, though): - pypiserver supports uploading to private indexes, devpi-server not yet (but next week / on trunk already :) - pypiserver has no dependencies (ships bottle inline), devpi-server depends on redis (to be dropped next week for no-dep fs-storage) and bottle, requests, py, all proven libraries. - pypiserver redirects the lookup of pypi packages to pypi.python.org, devpi-server caches them and serves everything (including #egg-links) through itself, allowing complete offline operations (including caching/serving of 3rd party site's packages) using the extended PEP381 mirror protocol - pypiserver has a good and simple implementation, devpi-server is little more complex mostly due to its caching/crawling logic. - both are very well tested and maintained but pypiserver is out there for a longer time already, so has seen more RL-testing. Ralph, please add/comment as you see fit. best, holger > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > From ronaldoussoren at mac.com Thu May 30 10:41:00 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 May 2013 10:41:00 +0200 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <20130529183633.GM7789@merlinux.eu> References: <20130529183633.GM7789@merlinux.eu> Message-ID: On 29 May, 2013, at 20:36, holger krekel wrote: > Hi Nick, > > On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: >> After preliminary reviews by Donald and Daniel, I have now pushed the >> first complete draft of the JSON-based metadata 2.0 proposal to >> python.org >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > After a first quick read i am wondering if i missed something > with respect to test tools. There are some fields which specify > dependencies required for running tests, but there is nothing that > would tell which test runner to use, which test command to invoke, > or am i missing something? > > Basically i am wondering how PEP426 could be useful/used by tox. > best, There is something about testing in the metabuild hooks section, although that appears to be targetted towards running tests for an installed distribution. I'm not sure yet how useful that hook would be for my packages, as the test suite isn't actually installed (it is in the sdist, but not in the output of bdist_*). Ronald From techtonik at gmail.com Thu May 30 12:01:27 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 30 May 2013 13:01:27 +0300 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: Message-ID: On Mon, May 27, 2013 at 3:08 AM, Donald Stufft wrote: > Hello! > > As you have have noticed the download counts on PyPI are no longer > updating. Originally this was due to an issue with the script that > processes these download counts. However I have now removed the download > counts from the PyPI webui and their use via the API is considered > deprecated. > This was the only motivation to continue supporting my packages. :( Of course it was an illusion that these were useful to someone, but it was so sweet. > There are numerous reasons for their removal/deprecation some of which are: > - Technically hard to make work with the new CDN > - The CDN is being donated to the PSF, and the donated tier does > not offer any form of log access > - The work around for not having log access would greatly reduce > the utility of the CDN > I don't believe that CDN clients don't want access to download stats - it is an essential feature for measuring performance and rates of any download service. Who is this provider who doesn't support them? > - Highly inaccurate > - A number of things prevent the download counts from being > inaccurate, some of which include: > - pip download cache > - Internal or unofficial mirrors > - Packages not hosted on PyPI (for comparisons sake) > - Mirrors or unofficial grab scripts causing inflated counts > (Last I looked 25% of the downloads were from a known mirroring script). > For less popular packages these factors are not that important. Also the exact count of human downloads is rarely interesting. Also everybody realizes there is no guarantee that download rate is not inflated. And still these counts provide good enough overview of relative package popularity. I wouldn't say that counts are highly inaccurate. For relative comparisons they are sane enough. Having inaccurate stats is much better than not having stats at all. Exposing download counts with a note about their accuracy will increase chances that people will be interested to work on improving the stats. > - Not particularly useful > - Just because a project has been downloaded a lot doesn't mean > it's good > - Similarly just because a project hasn't been downloaded a lot > doesn't mean it's bad > How about download count for a package released 7 years ago? The download count proves it is useful. > In short because it's value is low for various reasons, and the tradeoffs > required to make it work are high It has been not an effective use of > resources. > What are the tradeoffs? I'd like to preserve counts - that's why I got here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Thu May 30 13:53:53 2013 From: aclark at aclark.net (Alex Clark) Date: Thu, 30 May 2013 11:53:53 +0000 (UTC) Subject: [Distutils] PyPI Download Counts References: Message-ID: Hi, anatoly techtonik gmail.com> writes: > > On Mon, May 27, 2013 at 3:08 AM, Donald Stufft stufft.io> wrote: > > Hello! > > > As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated. > > > > This was the only motivation to continue supporting my packages. :( > Of course it was an illusion that these were useful to someone, but it was so sweet. +1-ish > > > ? > > There are numerous reasons for their removal/deprecation some of which are: > > ? ? - Technically hard to make work with the new CDN > ? ? ? ? - The CDN is being donated to the PSF, and the donated tier does not offer any form of log access > ? ? ? ? - The work around for not having log access would greatly reduce the utility of the CDN > > > > I don't believe that CDN clients don't want access to download stats - it is an essential feature for measuring performance and rates of any download service. Who is this provider who doesn't support them? > > ? > > ? ? - Highly inaccurate > > ? ? ? ? - A number of things prevent the download counts from being inaccurate, some of which include: > ? ? ? ? ? ? - pip download cache > ? ? ? ? ? ? - Internal or unofficial mirrors > ? ? ? ? ? ? - Packages not hosted on PyPI (for comparisons sake) > > ? ? ? ? ? ? - Mirrors or unofficial grab scripts causing inflated counts (Last I looked 25% of the downloads were from a known mirroring script). > > > For less popular packages these factors are not that important. Also the exact count of human downloads is rarely interesting. Also everybody realizes there is no guarantee that download rate is not inflated. And still these counts provide good enough overview of relative package popularity. > > > I wouldn't say that counts are highly inaccurate. For relative comparisons they are sane enough. > > Having inaccurate stats is much better than not having stats at all. Exposing download counts with a note about their accuracy will increase chances that people will be interested to work on improving the stats. > > ? > > > ? ? - Not particularly useful > > ? ? ? ? - Just because a project has been downloaded a lot doesn't mean it's good > ? ? ? ? - Similarly just because a project hasn't been downloaded a lot doesn't mean it's bad > > > > > How about download count for a package released 7 years ago? The download count proves it is useful. Well for now at least we have the history. You can use `vanity` [1] to get those stats... and graph them against zero, for today ;-) > ? > > > In short because it's value is low for various reasons, and the tradeoffs required to make it work are high It has been not an effective use of resources. > > > > What are the tradeoffs? I'd like to preserve counts - that's why I got here. I think there is a large enough group of folks that have chimed in here already to indicate that losing download stats is not entirely acceptable, but certainly a reasonable trade off when someone volunteers to do a lot of important and necessary hard work to improve our overall infrastructure :-). I'm sure we'll be able to re-add them at some point, but personally I'm going to let the CDN dust settle, and be thankful these folks are volunteering their time to do all the work (Donald, Noah, et al, thank you!) Alex > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG python.org > http://mail.python.org/mailman/listinfo/distutils-sig > --- Alex Clark * http://about.me/alex.clark From jim at zope.com Thu May 30 14:05:28 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 30 May 2013 08:05:28 -0400 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130527072731.GX7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> Message-ID: On Mon, May 27, 2013 at 3:27 AM, holger krekel wrote: > Hi Donald, > > On Sun, May 26, 2013 at 20:08 -0400, Donald Stufft wrote: >> Hello! >> >> As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated. >> >> There are numerous reasons for their removal/deprecation some of which are: >> - Technically hard to make work with the new CDN >> - The CDN is being donated to the PSF, and the donated tier does not offer any form of log access > > What would be involved money/effort wise to get such access? I didn't see an answer to this. Any idea how much this would cost? With access to logs, we could compute download counts. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From brian at vanguardistas.net Thu May 30 14:54:18 2013 From: brian at vanguardistas.net (Brian Sutherland) Date: Thu, 30 May 2013 14:54:18 +0200 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: Message-ID: <20130530125418.GA59528@Brians-MacBook-Air.local> On Sun, May 26, 2013 at 08:08:35PM -0400, Donald Stufft wrote: > Hello! > > As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated. > > There are numerous reasons for their removal/deprecation some of which are: > - Technically hard to make work with the new CDN > - The CDN is being donated to the PSF, and the donated tier does not offer any form of log access > - The work around for not having log access would greatly reduce the utility of the CDN > - Highly inaccurate > - A number of things prevent the download counts from being inaccurate, some of which include: > - pip download cache > - Internal or unofficial mirrors > - Packages not hosted on PyPI (for comparisons sake) > - Mirrors or unofficial grab scripts causing inflated counts (Last I looked 25% of the downloads were from a known mirroring script). > - Not particularly useful > - Just because a project has been downloaded a lot doesn't mean it's good > - Similarly just because a project hasn't been downloaded a lot doesn't mean it's bad > > In short because it's value is low for various reasons, and the tradeoffs required to make it work are high It has been not an effective use of resources. > > The API will continue to return values for it in order to not break scripts, however in the future all these values will be set to 0. The Web UI has been modified to no longer display it. There are other ways to get usage statistics. Debian has Popularity Contest (http://popcon.debian.org) statistics for python libraries. e.g. babel and twisted: http://qa.debian.org/popcon.php?package=python-babel http://qa.debian.org/popcon.php?package=twisted Of course, the these are probably still "Highly inaccurate", just in a different way;) -- Brian Sutherland From kiorky at cryptelium.net Thu May 30 17:08:05 2013 From: kiorky at cryptelium.net (kiorky) Date: Thu, 30 May 2013 17:08:05 +0200 Subject: [Distutils] problems with sdist upload since CDN update Message-ID: <51A76B55.5030207@cryptelium.net> I got some random releasing (1 on 5 or 6 tries) problems since last pypi update. For example, with collective.ckeditor (3.6.10 tag on github), the sdist upload will randomly not upload the file without reporting any error on console: https://github.com/collective/collective.ckeditor/tree/3.6.10 -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF Pensez ? l?environnement. N?imprimez ce courriel que si vous en avez vraiment besoin. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 263 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Thu May 30 17:56:43 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 May 2013 11:56:43 -0400 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> Message-ID: <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> On May 30, 2013, at 8:05 AM, Jim Fulton wrote: > On Mon, May 27, 2013 at 3:27 AM, holger krekel wrote: >> Hi Donald, >> >> On Sun, May 26, 2013 at 20:08 -0400, Donald Stufft wrote: >>> Hello! >>> >>> As you have have noticed the download counts on PyPI are no longer updating. Originally this was due to an issue with the script that processes these download counts. However I have now removed the download counts from the PyPI webui and their use via the API is considered deprecated. >>> >>> There are numerous reasons for their removal/deprecation some of which are: >>> - Technically hard to make work with the new CDN >>> - The CDN is being donated to the PSF, and the donated tier does not offer any form of log access >> >> What would be involved money/effort wise to get such access? > > I didn't see an answer to this. > > Any idea how much this would cost? With access to logs, we could > compute download counts. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton Fastly has given us access to their streaming log support. Infrastructure will be setting up a secure method for receiving these logs at which point PyPI can use them. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 30 18:13:55 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 May 2013 12:13:55 -0400 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: <51A76B55.5030207@cryptelium.net> References: <51A76B55.5030207@cryptelium.net> Message-ID: <6A8A4112-BFBD-44CF-8745-FE83FD56B539@stufft.io> On May 30, 2013, at 11:08 AM, kiorky wrote: > I got some random releasing (1 on 5 or 6 tries) problems since last pypi update. > For example, with collective.ckeditor (3.6.10 tag on github), the sdist upload will randomly not upload the file without reporting any error on console: > > > https://github.com/collective/collective.ckeditor/tree/3.6.10 > -- > Cordialement, > KiOrKY > GPG Key FingerPrint: 0x1A1194B7681112AF > Pensez ? l?environnement. > N?imprimez ce courriel que si vous en avez vraiment besoin. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Are you able to reproduce this still? Looking for something in the logs but I don't see anything. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Thu May 30 18:35:23 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 30 May 2013 18:35:23 +0200 Subject: [Distutils] PyPI Download Counts In-Reply-To: <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> Message-ID: On Thu, May 30, 2013 at 5:56 PM, Donald Stufft wrote: > Fastly has given us access to their streaming log support. > > Infrastructure will be setting up a secure method for receiving these logs > at which point PyPI can use them. Excellent! Awesomeness unfolds like a FantomenK beat! //Lennart From kiorky at cryptelium.net Thu May 30 19:20:13 2013 From: kiorky at cryptelium.net (kiorky) Date: Thu, 30 May 2013 19:20:13 +0200 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: <6A8A4112-BFBD-44CF-8745-FE83FD56B539@stufft.io> References: <51A76B55.5030207@cryptelium.net> <6A8A4112-BFBD-44CF-8745-FE83FD56B539@stufft.io> Message-ID: <51A78A4D.2050709@cryptelium.net> As i said, not reproducible, i did it too times this afternoon, but i cant reproduced it afterwhile. I didnt notice the upload problem at first but so no targz on pypi even of the 200 OK upload, i reupload, got it on pypi. So in the pypi admin, i did: - delete (EVIL TM) the sdist, reupload it, no problem. - Redelete, got problem (with 200 ok still). - Redelete, upload no problem sdist was there on pypi, and from then, i cant suceed in refalling it after that, even if i tried to for like 10 times. Useless to say that it was with the same local dist. I got troubles, that's no doubt; maybe network shadows at that time. So well, we will see if i can reproduce when time comes from a new release of my addons which will get again into problems. On 30/05/2013 18:13, Donald Stufft wrote: > > On May 30, 2013, at 11:08 AM, kiorky > wrote: > >> I got some random releasing (1 on 5 or 6 tries) problems since last pypi update. >> For example, with collective.ckeditor (3.6.10 tag on github), the sdist >> upload will randomly not upload the file without reporting any error on console: >> >> >> https://github.com/collective/collective.ckeditor/tree/3.6.10 >> -- >> Cordialement, >> KiOrKY >> GPG Key FingerPrint: 0x1A1194B7681112AF >> Pensez ? l?environnement. >> N?imprimez ce courriel que si vous en avez vraiment besoin. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> http://mail.python.org/mailman/listinfo/distutils-sig > > Are you able to reproduce this still? Looking for something in the logs but I > don't see anything. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF Pensez ? l?environnement. N?imprimez ce courriel que si vous en avez vraiment besoin. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 263 bytes Desc: OpenPGP digital signature URL: From holger at merlinux.eu Thu May 30 20:16:14 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 30 May 2013 18:16:14 +0000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: <20130529183633.GM7789@merlinux.eu> <20130529193348.GN7789@merlinux.eu> Message-ID: <20130530181614.GR7789@merlinux.eu> On Wed, May 29, 2013 at 16:26 -0400, Daniel Holth wrote: > On Wed, May 29, 2013 at 3:33 PM, holger krekel wrote: > > On Wed, May 29, 2013 at 15:15 -0400, Daniel Holth wrote: > >> On Wed, May 29, 2013 at 2:36 PM, holger krekel wrote: > >> > Hi Nick, > >> > > >> > On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: > >> >> After preliminary reviews by Donald and Daniel, I have now pushed the > >> >> first complete draft of the JSON-based metadata 2.0 proposal to > >> >> python.org > >> >> > >> >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > >> > > >> > After a first quick read i am wondering if i missed something > >> > with respect to test tools. There are some fields which specify > >> > dependencies required for running tests, but there is nothing that > >> > would tell which test runner to use, which test command to invoke, > >> > or am i missing something? > >> > > >> > Basically i am wondering how PEP426 could be useful/used by tox. > >> > best, > >> > >> The first thing we might do is to have setuptools expose its > >> test_suite argument as "extensions" : { "setuptools": { "test_suite": > >> "callable.name" } }. > > > > The way tox specifies testing is to allow arbitrary test commands not > > just unittest test suites. People use py.test, make and whatnot > > to run tests. > > tox could shove a JSON version of its config into the metadata as an > extension. > > If "something wrapped in a unittest suite" isn't the interface for > running tests (in the most boring way possible, with better runners > for the developers not the users of the package), then what should it > be? An entry point or way to execute a command line script, either coming from an test_requires dependency, or a script contained in the package. Note that py.test can run unittest test suites, but pytest test suites are not modeled as a unittest test suite. The same would be true if you use a test script contained in a package or "make" for that matter. IOW, i think there should be a field denoting which command line tool or script to run for the tests. If it's not defined and a test suite is defined, it's of course fine to invoke unitest's machinery on it. > >> I think we need the next version of the metadata or sdist 2.0 to > >> really do a better job than just running "setup.py test"; right now > >> the tests usually only make sense in the context of an unpacked sdist. > >> Input appreciated. > > > > We certainly don't want to advocate using "setup.py" for running tests as > > we want to get way from the neccessity for this file to exist (correct me > > if i am wrong with this presumption). The metadata should be rich > > enough to support tools like tox to perform the testing, much > > like pip for installations. > > The first priority is to get rid of "setup.py install" since that is > where most of the trouble comes from. It's not that big of a task to > relegate setup.py to its proper place as a build script. At the same > time it will eventually become possible to use non-distutils-derived > build tools without having to emulate a distutils setup.py. > > Wouldn't we continue to type tox, or nose, or py.test, with the > standard hook being mainly for automated tools that just want to > verify that something still works before building / installing? Typically projects depend on the test runner/framework for executing their tests, be it at development or pre-installation time. holger From holger at merlinux.eu Thu May 30 20:19:18 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 30 May 2013 18:19:18 +0000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: <20130529183633.GM7789@merlinux.eu> Message-ID: <20130530181918.GS7789@merlinux.eu> On Thu, May 30, 2013 at 10:41 +0200, Ronald Oussoren wrote: > On 29 May, 2013, at 20:36, holger krekel wrote: > > > Hi Nick, > > > > On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: > >> After preliminary reviews by Donald and Daniel, I have now pushed the > >> first complete draft of the JSON-based metadata 2.0 proposal to > >> python.org > >> > >> PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > > > After a first quick read i am wondering if i missed something > > with respect to test tools. There are some fields which specify > > dependencies required for running tests, but there is nothing that > > would tell which test runner to use, which test command to invoke, > > or am i missing something? > > > > Basically i am wondering how PEP426 could be useful/used by tox. > > best, > > There is something about testing in the metabuild hooks section, although that appears to be targetted towards running tests for an installed distribution. > > I'm not sure yet how useful that hook would be for my packages, as the test suite isn't actually installed (it is in the sdist, but not in the output of bdist_*). That's indeed another issue. In order to run tests you sometimes need to install a test package for it. I'd like to add this to tox but have refrained so far because of the involved packaging activities. Ideally, building+package+upload should be able to create both the actual package and the test package in case someone wants to run the tests. That wouldn't easily work for models where tests are put into subdirectories unless maybe the new python3 namespaces are used. best, holger > Ronald > From holger at merlinux.eu Thu May 30 20:28:20 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 30 May 2013 18:28:20 +0000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: Message-ID: <20130530182820.GT7789@merlinux.eu> Hi Nick, I am actually missing a "goals" section in this PEP. Who can/should use the PEP's new formats and definitions? Is it meant to fit well for python2.6, PyPy, Jython, etc? Also, what the intention/roadmap for adoption, how can existing and new tools deald with or distinguish old-style and PEP426 packaging semantics? I think such "process" aspects would provide very valuable context for the many players in the field. best, holger On Mon, May 27, 2013 at 21:36 +1000, Nick Coghlan wrote: > After preliminary reviews by Donald and Daniel, I have now pushed the > first complete draft of the JSON-based metadata 2.0 proposal to > python.org > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/ > > With the rationale and commentary, they're over 3000 lines between > them, so I'm not attaching them here. > > The rationale for many of the changes is at the end of each PEP, along > with some comments on features that I have either rejected or > deliberately chosen to defer to the next revision of the metadata (at > the earliest). > > Those with BitBucket accounts may also comment inline on the drafts here: > > PEP 426: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0426.txt?at=default > PEP 440: https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0440.txt?at=default > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From dholth at gmail.com Fri May 31 04:26:45 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 May 2013 22:26:45 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <20130530182820.GT7789@merlinux.eu> References: <20130530182820.GT7789@merlinux.eu> Message-ID: On Thu, May 30, 2013 at 2:28 PM, holger krekel wrote: > Hi Nick, > > I am actually missing a "goals" section in this PEP. Who can/should use > the PEP's new formats and definitions? Is it meant to fit well for > python2.6, PyPy, Jython, etc? Also, what the intention/roadmap for > adoption, how can existing and new tools deald with or distinguish > old-style and PEP426 packaging semantics? I think such "process" aspects > would provide very valuable context for the many players in the field. > > best, > > holger I agree that this pep says a lot about how it's different than previous metadata but doesn't seem to have a clear single purpose. We hope to accomplish a lot of things, but this PEP is just concerned with defining an interchange format for the data we already have in [setuptools-based] setup.py, and some additional things we think we will need to get where we want to go. In particular this PEP does not say anything about entirely replacing setup.py.. What Metadata 2.0 does help to do is to represent a lot more of the information that's locked up in setup.py (or obfuscated by coming out different ways depending on the system) so that other tools can deal with it instead of necessarily having setup.py take charge of everything from development to installation. Along with wheel and the filesystem layout (.dist-info directories) we aim to have a clean, well documented, static interface between development/build and install, making it much easier to work on one half of the problem or the other. A future meta-build system will define an API or interface for the things that setup.py does. Existing setup.py scripts will be only the first available implementation. The format is designed to work with all current versions and implementations of Python. The end user shouldn't really notice anything different until some number of packages stop using setup.py. The key insight this time is that "setup.py install" is more harmful than other kinds of setup.py and is the first thing we should get rid of. No one actually parses PKG-INFO so I expect the Metadata 2.0 transition to be relatively painless. The semantics are intended to be as backwards compatible as possible. A future version of setuptools will be able to produce and consume the new and old formats, and distlib provides an alternate consumer. Daniel Holth From setuptools at bugs.python.org Fri May 31 11:32:12 2013 From: setuptools at bugs.python.org (Keith Yang) Date: Fri, 31 May 2013 09:32:12 +0000 Subject: [Distutils] [issue149] 'pkg_resources' is not defined in ssl_support.py Message-ID: <1369992732.3.0.203572550331.issue149@psf.upfronthosting.co.za> New submission from Keith Yang: File "/Users/keitheis/Quest/biienv/lib/python2.7/site-packages/setuptools/package_index.py", line 161, in __init__ if verify_ssl and ssl_support.is_available and (ca_bundle or ssl_support.find_ca_bundle()): File "/Users/keitheis/Quest/biienv/lib/python2.7/site-packages/setuptools/ssl_support.py", line 239, in find_ca_bundle return pkg_resources.resource_filename('certifi', 'cacert.pem') NameError: global name 'pkg_resources' is not defined ---------- files: patch_ssl_support.diff keyword: pkg_resources messages: 717 nosy: keitheis priority: bug status: unread title: 'pkg_resources' is not defined in ssl_support.py Added file: http://bugs.python.org/setuptools/file89/patch_ssl_support.diff _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- Index: setuptools/ssl_support.py =================================================================== --- setuptools/ssl_support.py (revision 88997) +++ setuptools/ssl_support.py (working copy) @@ -236,7 +236,8 @@ if os.path.isfile(cert_path): return cert_path try: - return pkg_resources.resource_filename('certifi', 'cacert.pem') + from pkg_resources import resource_filename + return resource_filename('certifi', 'cacert.pem') except (ImportError, ResolutionError, ExtractionError): return None From sheep at sheep.art.pl Fri May 31 12:57:53 2013 From: sheep at sheep.art.pl (Radomir Dopieralski) Date: Fri, 31 May 2013 12:57:53 +0200 Subject: [Distutils] A process for removal of PyPi entries Message-ID: Hello, is there a defined process for removing useless entries from PyPi? I was looking for a name for a new project, and as a part of that, I searched on the Python Package Index to see if the names I came up with are not taken already. I stumbled upon this: https://pypi.python.org/pypi/fun/1.0.0 Please note that there is absolutely no information about this entry. There is no way to contact the author and ask him if he would be willing to give up that name, no website, not even a license or description. If you look into the uploaded source code, you will see that it's just a "hello world" program. That's not a problem for me, I just picked a different name for my package. Even if I wanted to use that name, I could add a prefix or suffix to it, to make it unique. But then I looked through the list of the entires and checked the ones that had no description or their description was suspicious. Just with the letter A I got: https://pypi.python.org/pypi/42/0 https://pypi.python.org/pypi/Aaronyoungnester/1.4.0 https://pypi.python.org/pypi/ABC/0.0.0 https://pypi.python.org/pypi/abhi/2.0.0 https://pypi.python.org/pypi/acme.sql/0.0.0 https://pypi.python.org/pypi/affix/1.0 https://pypi.python.org/pypi/agg2567/1.1.0 https://pypi.python.org/pypi/agg2567/1.1.0 https://pypi.python.org/pypi/airstream/0 https://pypi.python.org/pypi/ajl_nester/1.1.0 https://pypi.python.org/pypi/akali/1.3.0 https://pypi.python.org/pypi/alexis/0.1 https://pypi.python.org/pypi/amoi/.lol. https://pypi.python.org/pypi/aodag3/1.0.0 https://pypi.python.org/pypi/arch/0.0.1 https://pypi.python.org/pypi/arounded/0.0 https://pypi.python.org/pypi/AthleteClass/1.0.0 https://pypi.python.org/pypi/athletelist/1.1.0 https://pypi.python.org/pypi/athletelist_jw/1.0.0 https://pypi.python.org/pypi/athletelistlogan/1.3.0 https://pypi.python.org/pypi/atool/1.0.0 https://pypi.python.org/pypi/awesomeness/0.0.1 https://pypi.python.org/pypi/aws-cli/0.0 https://pypi.python.org/pypi/ayame/0.0 All of those entries share some properties: * no author and no way to contact the author * no website, website offline or obviously not related (like google.com) * no description or meaningless description * no download url or uploaded code, or the code that is uploaded is just a "hello world" or similar exercise * no license I think that all those properties, taken together, in practice mean that the particular entry is completely useless both to the Python community and to its author -- possibly being just an abandoned test. I also think that there should be a defined process for requesting of removal of such entries -- so that an actually useful project can take their place. I understand that some of those entries are placeholders for projects that are actively being worked upon, just not much is disclosed yet. In that case, they could at least have an author contact information, a link to the repository or an "development status: in planning" trove classifier. Those project would be left alone. An additional check that could be done on the PyPi side is whether the same PuPi user has also some other entries that are perhaps more meaningful and contain contact information. They could be then contacted and asked about the status of their abandoned entries. If such a process existed and was publicly announced in the PyPi documentation, then there would be less work with this kind of maintenance, and no animosity in case of a needed entry being removed -- people would know what to expect. Thank you for all the good work on PyPi, -- Radomir Dopieralski, http://sheep.art.pl From regebro at gmail.com Fri May 31 15:04:41 2013 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 31 May 2013 15:04:41 +0200 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: Message-ID: I support this message. On Fri, May 31, 2013 at 12:57 PM, Radomir Dopieralski wrote: > Hello, > > is there a defined process for removing useless entries from PyPi? > > I was looking for a name for a new project, and as a part of that, I searched > on the Python Package Index to see if the names I came up with are not taken > already. I stumbled upon this: > > https://pypi.python.org/pypi/fun/1.0.0 > > Please note that there is absolutely no information about this entry. There > is no way to contact the author and ask him if he would be willing to give > up that name, no website, not even a license or description. If you look into > the uploaded source code, you will see that it's just a "hello world" program. > > That's not a problem for me, I just picked a different name for my package. > Even if I wanted to use that name, I could add a prefix or suffix to it, to make > it unique. But then I looked through the list of the entires and checked the > ones that had no description or their description was suspicious. Just with the > letter A I got: > > https://pypi.python.org/pypi/42/0 > https://pypi.python.org/pypi/Aaronyoungnester/1.4.0 > https://pypi.python.org/pypi/ABC/0.0.0 > https://pypi.python.org/pypi/abhi/2.0.0 > https://pypi.python.org/pypi/acme.sql/0.0.0 > https://pypi.python.org/pypi/affix/1.0 > https://pypi.python.org/pypi/agg2567/1.1.0 > https://pypi.python.org/pypi/agg2567/1.1.0 > https://pypi.python.org/pypi/airstream/0 > https://pypi.python.org/pypi/ajl_nester/1.1.0 > https://pypi.python.org/pypi/akali/1.3.0 > https://pypi.python.org/pypi/alexis/0.1 > https://pypi.python.org/pypi/amoi/.lol. > https://pypi.python.org/pypi/aodag3/1.0.0 > https://pypi.python.org/pypi/arch/0.0.1 > https://pypi.python.org/pypi/arounded/0.0 > https://pypi.python.org/pypi/AthleteClass/1.0.0 > https://pypi.python.org/pypi/athletelist/1.1.0 > https://pypi.python.org/pypi/athletelist_jw/1.0.0 > https://pypi.python.org/pypi/athletelistlogan/1.3.0 > https://pypi.python.org/pypi/atool/1.0.0 > https://pypi.python.org/pypi/awesomeness/0.0.1 > https://pypi.python.org/pypi/aws-cli/0.0 > https://pypi.python.org/pypi/ayame/0.0 > > All of those entries share some properties: > > * no author and no way to contact the author > * no website, website offline or obviously not related (like google.com) > * no description or meaningless description > * no download url or uploaded code, or the code that is uploaded is just > a "hello world" or similar exercise > * no license > > I think that all those properties, taken together, in practice mean that the > particular entry is completely useless both to the Python community and to its > author -- possibly being just an abandoned test. I also think that there should > be a defined process for requesting of removal of such entries -- so that an > actually useful project can take their place. > > I understand that some of those entries are placeholders for projects that are > actively being worked upon, just not much is disclosed yet. In that case, they > could at least have an author contact information, a link to the repository or > an "development status: in planning" trove classifier. Those project would be > left alone. > > An additional check that could be done on the PyPi side is whether the same > PuPi user has also some other entries that are perhaps more meaningful and > contain contact information. They could be then contacted and asked about the > status of their abandoned entries. > > If such a process existed and was publicly announced in the PyPi documentation, > then there would be less work with this kind of maintenance, and no animosity > in case of a needed entry being removed -- people would know what to expect. > > Thank you for all the good work on PyPi, > > > -- > Radomir Dopieralski, http://sheep.art.pl > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From lukasz at langa.pl Fri May 31 15:06:41 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 31 May 2013 15:06:41 +0200 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: Message-ID: <2ECC5EC0-DFF6-4C5F-9080-DF73A8AC8D1F@langa.pl> On 31 maj 2013, at 12:57, Radomir Dopieralski wrote: > I was looking for a name for a new project, and as a part of that, I searched > on the Python Package Index to see if the names I came up with are not taken > already. I concur. It's increasingly easy to find bogus entries on the index. We've had this quite recently with Hynek trying to publish "first" and finding out there's a package of that name. Fortunately we were able to work it out with Richard but we had to contact him directly and waste his cycles on this. Same thing happened with "proxy" but in this case Richard decided that while the currect package is bogus, the name is bad anyway and he's not freeing it up ;) Which is fair enough, but again - wasted cycles and no clear process. > All of those entries share some properties: > > * no author and no way to contact the author > * no website, website offline or obviously not related (like google.com) > * no description or meaningless description > * no download url or uploaded code, or the code that is uploaded is just > a "hello world" or similar exercise > * no license The simplest approach would be to expire unused package names after, say, 6 months. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev From holger at merlinux.eu Fri May 31 15:09:39 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 31 May 2013 13:09:39 +0000 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: Message-ID: <20130531130939.GC7789@merlinux.eu> Hi Radomir, On Fri, May 31, 2013 at 12:57 +0200, Radomir Dopieralski wrote: > Hello, > > is there a defined process for removing useless entries from PyPi? > > I was looking for a name for a new project, and as a part of that, I searched > on the Python Package Index to see if the names I came up with are not taken > already. I stumbled upon this: > > https://pypi.python.org/pypi/fun/1.0.0 > > Please note that there is absolutely no information about this entry. There > is no way to contact the author and ask him if he would be willing to give > up that name, no website, not even a license or description. If you look into > the uploaded source code, you will see that it's just a "hello world" program. > > That's not a problem for me, I just picked a different name for my package. > Even if I wanted to use that name, I could add a prefix or suffix to it, to make > it unique. But then I looked through the list of the entires and checked the > ones that had no description or their description was suspicious. Just with the > letter A I got: > > https://pypi.python.org/pypi/42/0 > https://pypi.python.org/pypi/Aaronyoungnester/1.4.0 > https://pypi.python.org/pypi/ABC/0.0.0 > https://pypi.python.org/pypi/abhi/2.0.0 > https://pypi.python.org/pypi/acme.sql/0.0.0 > https://pypi.python.org/pypi/affix/1.0 > https://pypi.python.org/pypi/agg2567/1.1.0 > https://pypi.python.org/pypi/agg2567/1.1.0 > https://pypi.python.org/pypi/airstream/0 > https://pypi.python.org/pypi/ajl_nester/1.1.0 > https://pypi.python.org/pypi/akali/1.3.0 > https://pypi.python.org/pypi/alexis/0.1 > https://pypi.python.org/pypi/amoi/.lol. > https://pypi.python.org/pypi/aodag3/1.0.0 > https://pypi.python.org/pypi/arch/0.0.1 > https://pypi.python.org/pypi/arounded/0.0 > https://pypi.python.org/pypi/AthleteClass/1.0.0 > https://pypi.python.org/pypi/athletelist/1.1.0 > https://pypi.python.org/pypi/athletelist_jw/1.0.0 > https://pypi.python.org/pypi/athletelistlogan/1.3.0 > https://pypi.python.org/pypi/atool/1.0.0 > https://pypi.python.org/pypi/awesomeness/0.0.1 > https://pypi.python.org/pypi/aws-cli/0.0 > https://pypi.python.org/pypi/ayame/0.0 > > All of those entries share some properties: > > * no author and no way to contact the author > * no website, website offline or obviously not related (like google.com) > * no description or meaningless description > * no download url or uploaded code, or the code that is uploaded is just > a "hello world" or similar exercise > * no license > > I think that all those properties, taken together, in practice mean that the > particular entry is completely useless both to the Python community and to its > author -- possibly being just an abandoned test. I also think that there should > be a defined process for requesting of removal of such entries -- so that an > actually useful project can take their place. > > I understand that some of those entries are placeholders for projects that are > actively being worked upon, just not much is disclosed yet. In that case, they > could at least have an author contact information, a link to the repository or > an "development status: in planning" trove classifier. Those project would be > left alone. > > An additional check that could be done on the PyPi side is whether the same > PuPi user has also some other entries that are perhaps more meaningful and > contain contact information. They could be then contacted and asked about the > status of their abandoned entries. > > If such a process existed and was publicly announced in the PyPi documentation, > then there would be less work with this kind of maintenance, and no animosity > in case of a needed entry being removed -- people would know what to expect. I think there should be a PEP regulating the removal and the "taking over" process for packages. Your considerations make sense to me there. ASFAIK Perl has such policies a decade or so. Probably makes sense to use their provisions as a blue print. Such a PEP should contain a list of projects that are to be removed retro-actively if they don't comply within 6 months or so. I think the barrier shouldn't be too high, a valid email address and/or release activity are a good minimum. And it should be a short PEP :) cheers, holger > > Thank you for all the good work on PyPi, > > -- > Radomir Dopieralski, http://sheep.art.pl > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From regebro at gmail.com Fri May 31 15:18:30 2013 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 31 May 2013 15:18:30 +0200 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <20130531130939.GC7789@merlinux.eu> References: <20130531130939.GC7789@merlinux.eu> Message-ID: On Fri, May 31, 2013 at 3:09 PM, holger krekel wrote: > I think there should be a PEP regulating the removal and the "taking over" > process for packages. Your considerations make sense to me there. > ASFAIK Perl has such policies a decade or so. Probably makes sense to > use their provisions as a blue print. Such a PEP should contain a > list of projects that are to be removed retro-actively if they don't > comply within 6 months or so. I think the barrier shouldn't be too high, > a valid email address and/or release activity are a good minimum. > And it should be a short PEP :) I'd be OK with after six months automatically removing packages that has only one owner/maintainer, and that owner/maintainer has no other packages, and the package has no available downloads, and no contact information on either package nor registered user. For other cases there should also be a manual way or removing packages on the discretion of PyPI maintainers, as well as giving owner access to packages whose owner is unreachable (I think there may already exist a process for that). //Lennart From donald at stufft.io Fri May 31 16:54:41 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 31 May 2013 10:54:41 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: Message-ID: <0355BAFD-4BDD-48C8-9D45-2D806532D6E5@stufft.io> On May 31, 2013, at 6:57 AM, Radomir Dopieralski wrote: > Hello, > > is there a defined process for removing useless entries from PyPi? > > I was looking for a name for a new project, and as a part of that, I searched > on the Python Package Index to see if the names I came up with are not taken > already. I stumbled upon this: > > https://pypi.python.org/pypi/fun/1.0.0 > > Please note that there is absolutely no information about this entry. There > is no way to contact the author and ask him if he would be willing to give > up that name, no website, not even a license or description. If you look into > the uploaded source code, you will see that it's just a "hello world" program. > > That's not a problem for me, I just picked a different name for my package. > Even if I wanted to use that name, I could add a prefix or suffix to it, to make > it unique. But then I looked through the list of the entires and checked the > ones that had no description or their description was suspicious. Just with the > letter A I got: > > https://pypi.python.org/pypi/42/0 > https://pypi.python.org/pypi/Aaronyoungnester/1.4.0 > https://pypi.python.org/pypi/ABC/0.0.0 > https://pypi.python.org/pypi/abhi/2.0.0 > https://pypi.python.org/pypi/acme.sql/0.0.0 > https://pypi.python.org/pypi/affix/1.0 > https://pypi.python.org/pypi/agg2567/1.1.0 > https://pypi.python.org/pypi/agg2567/1.1.0 > https://pypi.python.org/pypi/airstream/0 > https://pypi.python.org/pypi/ajl_nester/1.1.0 > https://pypi.python.org/pypi/akali/1.3.0 > https://pypi.python.org/pypi/alexis/0.1 > https://pypi.python.org/pypi/amoi/.lol. > https://pypi.python.org/pypi/aodag3/1.0.0 > https://pypi.python.org/pypi/arch/0.0.1 > https://pypi.python.org/pypi/arounded/0.0 > https://pypi.python.org/pypi/AthleteClass/1.0.0 > https://pypi.python.org/pypi/athletelist/1.1.0 > https://pypi.python.org/pypi/athletelist_jw/1.0.0 > https://pypi.python.org/pypi/athletelistlogan/1.3.0 > https://pypi.python.org/pypi/atool/1.0.0 > https://pypi.python.org/pypi/awesomeness/0.0.1 > https://pypi.python.org/pypi/aws-cli/0.0 > https://pypi.python.org/pypi/ayame/0.0 > > All of those entries share some properties: > > * no author and no way to contact the author > * no website, website offline or obviously not related (like google.com) > * no description or meaningless description > * no download url or uploaded code, or the code that is uploaded is just > a "hello world" or similar exercise > * no license > > I think that all those properties, taken together, in practice mean that the > particular entry is completely useless both to the Python community and to its > author -- possibly being just an abandoned test. I also think that there should > be a defined process for requesting of removal of such entries -- so that an > actually useful project can take their place. > > I understand that some of those entries are placeholders for projects that are > actively being worked upon, just not much is disclosed yet. In that case, they > could at least have an author contact information, a link to the repository or > an "development status: in planning" trove classifier. Those project would be > left alone. > > An additional check that could be done on the PyPi side is whether the same > PuPi user has also some other entries that are perhaps more meaningful and > contain contact information. They could be then contacted and asked about the > status of their abandoned entries. > > If such a process existed and was publicly announced in the PyPi documentation, > then there would be less work with this kind of maintenance, and no animosity > in case of a needed entry being removed -- people would know what to expect. > > Thank you for all the good work on PyPi, > > > -- > Radomir Dopieralski, http://sheep.art.pl > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig There is no defined process. Getting one would be good. A PEP is likely warranted in order to define the process. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From maphew at gmail.com Fri May 31 19:44:19 2013 From: maphew at gmail.com (Matt Wilkie) Date: Fri, 31 May 2013 10:44:19 -0700 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: <51A76B55.5030207@cryptelium.net> References: <51A76B55.5030207@cryptelium.net> Message-ID: I'm also having trouble with uploading to pypi, it's not random, meaning it's happened every time so far. My last upload was about a week ago and seamless. Here is my best reconstruction of today: {{{ python setup.py sdist upload ...snip adding 'leo-4.11.devel-build-5802\leo.egg-info\PKG-INFO' adding 'leo-4.11.devel-build-5802\leo.egg-info\SOURCES.txt' adding 'leo-4.11.devel-build-5802\leo.egg-info\top_level.txt' removing 'leo-4.11.devel-build-5802' (and everything under it) running upload submitting d:/leo/dist\leo-4.11.devel-build-5802.zip to http://pypi.python.org/pypi upload failed (500): There's been a problem with your request python setup.py sdist upload :: same error python setup.py register :: was okay python setup.py sdist upload :: same error }}} I then went to https://pypi.python.org/pypi?%3Aaction=pkg_edit&name=leo and noticed that I had my role mysteriously expanded to include Owner as well as Maintainer. I used the web interface and removed the Owner role for myself, and then manually uploaded the sdist package, which succeeded. Went back to cmd shell and: {{{ B:\apps\leo\pypi-411>python setup.py bdist_wininst upload ...snip adding 'SCRIPTS\leoc-script.py' adding 'SCRIPTS\leoc.exe' adding 'SCRIPTS\leoc.exe.manifest' removing 'd:/leo/build' (and everything under it) running upload Submitting d:/leo/dist\leo-4.11.devel-build-5802.win32.exe to http://pypi.python.org/pypi Server response (200): OK Submitting d:/leo/dist\leo-4.11.devel-build-5802.win32.exe to http://pypi.python.org/pypi Upload failed (400): A file named "leo-4.11.devel-build-5802.win32.exe" already exists for leo-4.11.devel-build-5802. To fix problems with that file you should create a new release. }}} Note the double upload. Going back to the website https://pypi.python.org/pypi/leo there are now 2 packages listed: "leo 4.10-final", which corresponds to the sdist package, and "leo 4.11.devel-build-5802" corresponding the win32 .exe installer. The metadata for 4.10-final has the correct author and maintainer, but all the other metadata is old and hasn't been updated. The metadata for 4.11-devel is up to date for homepage through maintainer but has the wrong Author and is missing the long description. The journal entries for the 2 packages mirror each other except for the last line: {{{ Journal Action Date User Address create 2003-10-09 17:32 edreamleo 66.168.19.217 add Owner edreamleo 2003-10-09 17:32 edreamleo 66.168.19.217 new release 2012-03-29 13:53 edreamleo 68.185.171.138 add Maintainer maphew 2013-05-07 13:08 edreamleo 172.8.201.39 add Owner maphew 2013-05-22 23:21 maphew 199.247.128.35 update _pypi_hidden 2013-05-22 23:22 maphew 199.247.128.35 update _pypi_hidden 2013-05-31 16:52 maphew 199.27.75.22 remove Owner maphew 2013-05-31 16:53 maphew 199.27.75.22 update hosting_mode 2013-05-31 16:56 maphew 199.27.75.20 add source file leo-4.11.devel-build-5802.zip 2013-05-31 16:59 maphew 199.27.75.21 }}} {{{ add any file leo-4.11.devel-build-5802.win32.exe 2013-05-31 17:02 maphew 199.27.75.23 }}} Also curious is that all the ip addresses for today are wrong and variable. My external-to-world-ip should be 199.247.128.35 and static. cheers, -matt From donald at stufft.io Fri May 31 20:10:46 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 31 May 2013 14:10:46 -0400 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: References: <51A76B55.5030207@cryptelium.net> Message-ID: <7F3F8244-8940-4D30-9FC2-236297581A55@stufft.io> On May 31, 2013, at 1:44 PM, Matt Wilkie wrote: > I'm also having trouble with uploading to pypi, it's not random, > meaning it's happened every time so far. My last upload was about a > week ago and seamless. > > Here is my best reconstruction of today: > > {{{ > python setup.py sdist upload > > ...snip > adding 'leo-4.11.devel-build-5802\leo.egg-info\PKG-INFO' > adding 'leo-4.11.devel-build-5802\leo.egg-info\SOURCES.txt' > adding 'leo-4.11.devel-build-5802\leo.egg-info\top_level.txt' > removing 'leo-4.11.devel-build-5802' (and everything under it) > running upload > submitting d:/leo/dist\leo-4.11.devel-build-5802.zip to > http://pypi.python.org/pypi > upload failed (500): There's been a problem with your request > > python setup.py sdist upload > :: same error > > python setup.py register > :: was okay > > python setup.py sdist upload > :: same error > }}} > > I then went to https://pypi.python.org/pypi?%3Aaction=pkg_edit&name=leo > and noticed that I had my role mysteriously expanded to include Owner > as well as Maintainer. > > I used the web interface and removed the Owner role for myself, and > then manually uploaded the sdist package, which succeeded. > > Went back to cmd shell and: > > {{{ > B:\apps\leo\pypi-411>python setup.py bdist_wininst upload > > ...snip > adding 'SCRIPTS\leoc-script.py' > adding 'SCRIPTS\leoc.exe' > adding 'SCRIPTS\leoc.exe.manifest' > removing 'd:/leo/build' (and everything under it) > running upload > Submitting d:/leo/dist\leo-4.11.devel-build-5802.win32.exe to > http://pypi.python.org/pypi > Server response (200): OK > Submitting d:/leo/dist\leo-4.11.devel-build-5802.win32.exe to > http://pypi.python.org/pypi > Upload failed (400): A file named > "leo-4.11.devel-build-5802.win32.exe" already exists for > leo-4.11.devel-build-5802. To fix problems with that file you should > create a new release. > }}} > > Note the double upload. > > Going back to the website https://pypi.python.org/pypi/leo there are > now 2 packages listed: "leo 4.10-final", which corresponds to the > sdist package, and "leo 4.11.devel-build-5802" corresponding the win32 > .exe installer. > > The metadata for 4.10-final has the correct author and maintainer, but > all the other metadata is old and hasn't been updated. > > The metadata for 4.11-devel is up to date for homepage through > maintainer but has the wrong Author and is missing the long > description. > > The journal entries for the 2 packages mirror each other except for > the last line: > {{{ > Journal > Action Date User Address > create 2003-10-09 17:32 edreamleo 66.168.19.217 > add Owner edreamleo 2003-10-09 17:32 edreamleo 66.168.19.217 > new release 2012-03-29 13:53 edreamleo 68.185.171.138 > add Maintainer maphew 2013-05-07 13:08 edreamleo 172.8.201.39 > add Owner maphew 2013-05-22 23:21 maphew 199.247.128.35 > update _pypi_hidden 2013-05-22 23:22 maphew 199.247.128.35 > update _pypi_hidden 2013-05-31 16:52 maphew 199.27.75.22 > remove Owner maphew 2013-05-31 16:53 maphew 199.27.75.22 > update hosting_mode 2013-05-31 16:56 maphew 199.27.75.20 > add source file leo-4.11.devel-build-5802.zip 2013-05-31 16:59 > maphew 199.27.75.21 > }}} > > {{{ > add any file leo-4.11.devel-build-5802.win32.exe 2013-05-31 17:02 > maphew 199.27.75.23 > }}} > > Also curious is that all the ip addresses for today are wrong and > variable. My external-to-world-ip should be 199.247.128.35 and static. > > cheers, > > -matt > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I see an exception here in Sentry, I'll investigate further in a bit. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Fri May 31 21:44:49 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 31 May 2013 15:44:49 -0400 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: <7F3F8244-8940-4D30-9FC2-236297581A55@stufft.io> References: <51A76B55.5030207@cryptelium.net> <7F3F8244-8940-4D30-9FC2-236297581A55@stufft.io> Message-ID: <2AC2FE75-B60F-4AB9-92A1-F50001AC6039@stufft.io> On May 31, 2013, at 2:10 PM, Donald Stufft wrote: > > On May 31, 2013, at 1:44 PM, Matt Wilkie wrote: > >> I'm also having trouble with uploading to pypi, it's not random, >> meaning it's happened every time so far. My last upload was about a >> week ago and seamless. >> >> Here is my best reconstruction of today: >> >> {{{ >> python setup.py sdist upload >> >> ...snip >> adding 'leo-4.11.devel-build-5802\leo.egg-info\PKG-INFO' >> adding 'leo-4.11.devel-build-5802\leo.egg-info\SOURCES.txt' >> adding 'leo-4.11.devel-build-5802\leo.egg-info\top_level.txt' >> removing 'leo-4.11.devel-build-5802' (and everything under it) >> running upload >> submitting d:/leo/dist\leo-4.11.devel-build-5802.zip to >> http://pypi.python.org/pypi >> upload failed (500): There's been a problem with your request >> >> python setup.py sdist upload >> :: same error >> >> python setup.py register >> :: was okay >> >> python setup.py sdist upload >> :: same error >> }}} >> >> I then went to https://pypi.python.org/pypi?%3Aaction=pkg_edit&name=leo >> and noticed that I had my role mysteriously expanded to include Owner >> as well as Maintainer. >> >> I used the web interface and removed the Owner role for myself, and >> then manually uploaded the sdist package, which succeeded. >> >> Went back to cmd shell and: >> >> {{{ >> B:\apps\leo\pypi-411>python setup.py bdist_wininst upload >> >> ...snip >> adding 'SCRIPTS\leoc-script.py' >> adding 'SCRIPTS\leoc.exe' >> adding 'SCRIPTS\leoc.exe.manifest' >> removing 'd:/leo/build' (and everything under it) >> running upload >> Submitting d:/leo/dist\leo-4.11.devel-build-5802.win32.exe to >> http://pypi.python.org/pypi >> Server response (200): OK >> Submitting d:/leo/dist\leo-4.11.devel-build-5802.win32.exe to >> http://pypi.python.org/pypi >> Upload failed (400): A file named >> "leo-4.11.devel-build-5802.win32.exe" already exists for >> leo-4.11.devel-build-5802. To fix problems with that file you should >> create a new release. >> }}} >> >> Note the double upload. >> >> Going back to the website https://pypi.python.org/pypi/leo there are >> now 2 packages listed: "leo 4.10-final", which corresponds to the >> sdist package, and "leo 4.11.devel-build-5802" corresponding the win32 >> .exe installer. >> >> The metadata for 4.10-final has the correct author and maintainer, but >> all the other metadata is old and hasn't been updated. >> >> The metadata for 4.11-devel is up to date for homepage through >> maintainer but has the wrong Author and is missing the long >> description. >> >> The journal entries for the 2 packages mirror each other except for >> the last line: >> {{{ >> Journal >> Action Date User Address >> create 2003-10-09 17:32 edreamleo 66.168.19.217 >> add Owner edreamleo 2003-10-09 17:32 edreamleo 66.168.19.217 >> new release 2012-03-29 13:53 edreamleo 68.185.171.138 >> add Maintainer maphew 2013-05-07 13:08 edreamleo 172.8.201.39 >> add Owner maphew 2013-05-22 23:21 maphew 199.247.128.35 >> update _pypi_hidden 2013-05-22 23:22 maphew 199.247.128.35 >> update _pypi_hidden 2013-05-31 16:52 maphew 199.27.75.22 >> remove Owner maphew 2013-05-31 16:53 maphew 199.27.75.22 >> update hosting_mode 2013-05-31 16:56 maphew 199.27.75.20 >> add source file leo-4.11.devel-build-5802.zip 2013-05-31 16:59 >> maphew 199.27.75.21 >> }}} >> >> {{{ >> add any file leo-4.11.devel-build-5802.win32.exe 2013-05-31 17:02 >> maphew 199.27.75.23 >> }}} >> >> Also curious is that all the ip addresses for today are wrong and >> variable. My external-to-world-ip should be 199.247.128.35 and static. >> >> cheers, >> >> -matt >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > I see an exception here in Sentry, I'll investigate further in a bit. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Matt, is your code available online anywhere? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tseaver at palladion.com Fri May 31 22:34:43 2013 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 31 May 2013 16:34:43 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/31/2013 09:18 AM, Lennart Regebro wrote: > I'd be OK with after six months automatically removing packages that > has only one owner/maintainer, and that owner/maintainer has no other > packages, and the package has no available downloads, and no contact > information on either package nor registered user. Why all the extras: if somebody wants to claim a project name, but can't upload a release for six months, they should just lose. I would actually be willing to have that cut down to a day: trying to grab the name before registering / uploading a release should result in loss of the claim. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGpCWIACgkQ+gerLs4ltQ6e5wCbBm6t1T8a5ffPGBEFV0K3s3Fg jA8AoLVfTLCrSy6aCAbogcEouQc8H8Ak =bZTd -----END PGP SIGNATURE----- From noah at coderanger.net Fri May 31 22:45:04 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Fri, 31 May 2013 13:45:04 -0700 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> Message-ID: On May 31, 2013, at 1:34 PM, Tres Seaver wrote: > On 05/31/2013 09:18 AM, Lennart Regebro wrote: > > I'd be OK with after six months automatically removing packages that > > has only one owner/maintainer, and that owner/maintainer has no other > > packages, and the package has no available downloads, and no contact > > information on either package nor registered user. > > Why all the extras: if somebody wants to claim a project name, but can't > upload a release for six months, they should just lose. I would actually > be willing to have that cut down to a day: trying to grab the name > before registering / uploading a release should result in loss of the claim. +1, I think this should just be treated as a form validation thing. It is a detail of the protocol that you upload a dist definition before the files, but I don't think we should consider it a valid PyPI entry until a file is uploaded (especially now that the default mode is to not scrape external sites). As we switch to not scraping, anything with no files should just vanish IMO, at which point it is available for registration again. If someone happens to ninja-upload between the setup.py register and setup.py upload, I think we can just throw an error message since chances of that happening are so amazingly low. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Fri May 31 22:47:19 2013 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 31 May 2013 22:47:19 +0200 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> Message-ID: On Fri, May 31, 2013 at 10:34 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 05/31/2013 09:18 AM, Lennart Regebro wrote: >> I'd be OK with after six months automatically removing packages that >> has only one owner/maintainer, and that owner/maintainer has no other >> packages, and the package has no available downloads, and no contact >> information on either package nor registered user. > > Why all the extras: if somebody wants to claim a project name, but can't > upload a release for six months, they should just lose. I would actually > be willing to have that cut down to a day: trying to grab the name > before registering / uploading a release should result in loss of the claim. Because it's automatic. I don't trust computers. ;-) But yeah, it could be shorter. //Lennart From donald at stufft.io Fri May 31 22:49:21 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 31 May 2013 16:49:21 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> Message-ID: <16722708-4FCE-4959-8708-06C762E97AAD@stufft.io> On May 31, 2013, at 4:45 PM, Noah Kantrowitz wrote: > > On May 31, 2013, at 1:34 PM, Tres Seaver wrote: > >> On 05/31/2013 09:18 AM, Lennart Regebro wrote: >>> I'd be OK with after six months automatically removing packages that >>> has only one owner/maintainer, and that owner/maintainer has no other >>> packages, and the package has no available downloads, and no contact >>> information on either package nor registered user. >> >> Why all the extras: if somebody wants to claim a project name, but can't >> upload a release for six months, they should just lose. I would actually >> be willing to have that cut down to a day: trying to grab the name >> before registering / uploading a release should result in loss of the claim. > > +1, I think this should just be treated as a form validation thing. It is a detail of the protocol that you upload a dist definition before the files, but I don't think we should consider it a valid PyPI entry until a file is uploaded (especially now that the default mode is to not scrape external sites). As we switch to not scraping, anything with no files should just vanish IMO, at which point it is available for registration again. If someone happens to ninja-upload between the setup.py register and setup.py upload, I think we can just throw an error message since chances of that happening are so amazingly low. > > --Noah > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig So I completely agree with the sentiment. However we need to make sure whatever process we come up with has provisions for when it's ok to manually remove a name as well. The reasoning is that it can easily become an arms race of sort. If we say "well projects without a file get auto deleted after a day", then someone wanting to squat a name will just upload a dummy file. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tk47 at students.poly.edu Fri May 31 22:45:10 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Fri, 31 May 2013 16:45:10 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> Message-ID: <51A90BD6.5080801@students.poly.edu> On Fri 31 May 2013 04:34:43 PM EDT, Tres Seaver wrote: > > Why all the extras: if somebody wants to claim a project name, but can't > upload a release for six months, they should just lose. I would actually > be willing to have that cut down to a day: trying to grab the name > before registering / uploading a release should result in loss of the claim. > Firstly, let me say that the general idea sounds good, and should serve to improve PyPI security. However, it needs to be done carefully. Certainly Holger's idea of looking at how other programming language communities have done it is a good one. A potential problem with the "no new package in six months" heuristic is that it would punish mature packages with little or no improvements left. Would one defeat this rule by simply uploading a "new" package every six months? I am aware that packages have to change from time to time, if at least to keep up with language or other dependency changes, but the rules for weeding packages should be carefully thought out. From vinay_sajip at yahoo.co.uk Fri May 31 23:59:31 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 31 May 2013 21:59:31 +0000 (UTC) Subject: [Distutils] A process for removal of PyPi entries References: <20130531130939.GC7789@merlinux.eu> Message-ID: Tres Seaver palladion.com> writes: > Why all the extras: if somebody wants to claim a project name, but can't > upload a release for six months, they should just lose. I would actually > be willing to have that cut down to a day: trying to grab the name > before registering / uploading a release should result in loss of the claim. Six months does seem somewhat long, but I think a day is draconian. For example, I registered the PyPI name for "distil" when I first thought of the name (part way through developing it, but some weeks before the first release). The first release makes first impressions, so it makes sense to take care over it, and that takes time - sometimes, more than a few days. While I don't think people should be able to claim and park project names indefinitely, the choice of a project name is not totally trivial, and if a developer picks a name they like for one of their projects and it's available, I don't believe they should be forced to upload their first release too soon. This doesn't seem like an issue which requires precipitate measures. (I do agree with the general sentiments of tightening up on project names, and having a process to harvest claimed but unused project names after a decent interval.) Regards, Vinay Sajip From chris.barker at noaa.gov Fri May 31 22:17:05 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 31 May 2013 13:17:05 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X Message-ID: HI Folks, A few of us over on the pythonmac list have been hashing out how best to support binary packages on OS-X. The binary wheel option seems very promising. However, there is one Mac-specific issue that does not seem to be addressed: On OS-X, binaries can be "universal" -- what this means is that they have more than one binary architecture embedded in a single file; the system selects the right one at run time. Both executables and dynamic libraries can be universal. In theory, an universal binary can currently contain up to 4 architectures: 32+64 bit PPC and 32+64bit x86 In practice, 64 bit PPC was never broadly used, and 32bit PPC is fading fast. Nevertheless, both 32 and 64 Intel systems are pretty common, and who knows what there may be in the future (you never know with Apple...) The binary builds of Python served up on the python.org web site have supported universal builds for years -- currently there are two options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it turns out it's hard to support both up to date x86_64 and PPC with the same compiler. In these builds, the configuration is set up so that distutils will build universal extensions that match the architectures included in the builds. This makes it pretty easy to build binary packages, and/or full app distributions (py2app) that are universal as well. setuptools' easy_install supports binary eggs, but always choked on universal builds -- it didn't know how to identify what matched the system. It would be really nice if future tools could identify and install the correct binary wheels for universal builds. I've taken a look at PEP 427, and see this: platform tag E.g. 'linux_x86_64', 'any'. That looks to me like there is a built-in assumption that a wheel is either specific to a single architecture, or any -- so no way to specify that it may have multiple architectures. So I propose that platform tag allow a list of some sort: ['macosx-10_6_i386', 'macosx_10_6-x86_64'] or, I suppose to keep it simple, a single string: 'macosx_10_6_i386+macosx_10_6_x86_64' or 'macosx_10_6_i386+x86_64' If so, then a binary wheel could be built (and discovered) that supported multiple architectures. Then how would an installer ( pip? ) identify the right one? This is a bit tricky. PEP 425 states: "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore" On my system right now, I have 32bitPPC + 32bit i386 universal python, running on an intel system: In [2]: distutils.util.get_platform() Out[2]: 'macosx-10.3-i386' So we may want to patch get_platform() to support universal builds - or add anther function or something -- you may sometimes what to know what you are running right now, and other times want to know how the current python is built. (note that while distutils can build universal binaries, it's not very smart about it -- all is does is grab the compiler and linker flags from the Makefile, which include multiple "-arch" flags. Even if there is an easy way to query the system, that leaves the question of what to do. If a user wants to intall a package into a universal python, will the installer only accept a binary wheel with all the supported architectures? or one with only the currently running one? Anyway, it would be nice to be able to get this stuff to "just work" with universal binaries on the Mac. Thoughts, ideas? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov