From tk47 at students.poly.edu Sat Jun 1 00:33:09 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Fri, 31 May 2013 18:33:09 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51A90BD6.5080801@students.poly.edu> Message-ID: <51A92525.50307@students.poly.edu> On Fri 31 May 2013 06:16:28 PM EDT, Jim Fulton wrote: > > I think Tres was referring to the first release. > Thanks for the clarification, but my argument remains for subsequent releases. From dholth at gmail.com Sat Jun 1 00:38:30 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 31 May 2013 18:38:30 -0400 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: Wheel already supports compound tags. Just like py2.py3-none-any a tag py3-none-x86.ppc (with real platform names) would work. Does that make sense for Mac? On May 31, 2013 6:30 PM, "Chris Barker - NOAA Federal" < chris.barker at noaa.gov> wrote: > HI Folks, > > A few of us over on the pythonmac list have been hashing out how best > to support binary packages on OS-X. The binary wheel option seems very > promising. > > However, there is one Mac-specific issue that does not seem to be > addressed: > > On OS-X, binaries can be "universal" -- what this means is that they > have more than one binary architecture embedded in a single file; the > system selects the right one at run time. Both executables and dynamic > libraries can be universal. In theory, an universal binary can > currently contain up to 4 architectures: > > 32+64 bit PPC and 32+64bit x86 > > In practice, 64 bit PPC was never broadly used, and 32bit PPC is > fading fast. Nevertheless, both 32 and 64 Intel systems are pretty > common, and who knows what there may be in the future (you never know > with Apple...) > > The binary builds of Python served up on the python.org web site have > supported universal builds for years -- currently there are two > options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it > turns out it's hard to support both up to date x86_64 and PPC with the > same compiler. In these builds, the configuration is set up so that > distutils will build universal extensions that match the architectures > included in the builds. This makes it pretty easy to build binary > packages, and/or full app distributions (py2app) that are universal as > well. > > setuptools' easy_install supports binary eggs, but always choked on > universal builds -- it didn't know how to identify what matched the > system. It would be really nice if future tools could identify and > install the correct binary wheels for universal builds. > > I've taken a look at PEP 427, and see this: > > platform tag > E.g. 'linux_x86_64', 'any'. > > That looks to me like there is a built-in assumption that a wheel is > either specific to a single architecture, or any -- so no way to > specify that it may have multiple architectures. So I propose that > platform tag allow a list of some sort: > > ['macosx-10_6_i386', 'macosx_10_6-x86_64'] > > or, I suppose to keep it simple, a single string: > > 'macosx_10_6_i386+macosx_10_6_x86_64' > or > 'macosx_10_6_i386+x86_64' > > If so, then a binary wheel could be built (and discovered) that > supported multiple architectures. > > Then how would an installer ( pip? ) identify the right one? This is a > bit tricky. PEP 425 states: > > "The platform tag is simply distutils.util.get_platform() with all > hyphens - and periods . replaced with underscore" > > On my system right now, I have 32bitPPC + 32bit i386 universal python, > running on an intel system: > > In [2]: distutils.util.get_platform() > Out[2]: 'macosx-10.3-i386' > > So we may want to patch get_platform() to support universal builds - > or add anther function or something -- you may sometimes what to know > what you are running right now, and other times want to know how the > current python is built. (note that while distutils can build > universal binaries, it's not very smart about it -- all is does is > grab the compiler and linker flags from the Makefile, which include > multiple "-arch" flags. > > Even if there is an easy way to query the system, that leaves the > question of what to do. If a user wants to intall a package into a > universal python, will the installer only accept a binary wheel with > all the supported architectures? or one with only the currently > running one? > > Anyway, it would be nice to be able to get this stuff to "just work" > with universal binaries on the Mac. > > Thoughts, ideas? > > -Chris > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Sat Jun 1 00:41:36 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 31 May 2013 18:41:36 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <51A92525.50307@students.poly.edu> References: <20130531130939.GC7789@merlinux.eu> <51A90BD6.5080801@students.poly.edu> <51A92525.50307@students.poly.edu> Message-ID: On Fri, May 31, 2013 at 6:33 PM, Trishank Karthik Kuppusamy wrote: > On Fri 31 May 2013 06:16:28 PM EDT, Jim Fulton wrote: >> >> >> I think Tres was referring to the first release. >> > > Thanks for the clarification, but my argument remains for subsequent > releases. Hopefully, subsequent releases aren't an issue. IOW, I hope no one is proposing removing a project simply because someone hasn't released a new (after initial) release in some period of time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Sat Jun 1 00:16:28 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 31 May 2013 18:16:28 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <51A90BD6.5080801@students.poly.edu> References: <20130531130939.GC7789@merlinux.eu> <51A90BD6.5080801@students.poly.edu> Message-ID: On Fri, May 31, 2013 at 4:45 PM, Trishank Karthik Kuppusamy wrote: > On Fri 31 May 2013 04:34:43 PM EDT, Tres Seaver wrote: >> >> >> Why all the extras: if somebody wants to claim a project name, but can't >> upload a release for six months, they should just lose. I would actually >> be willing to have that cut down to a day: trying to grab the name >> before registering / uploading a release should result in loss of the >> claim. >> > > Firstly, let me say that the general idea sounds good, and should serve to > improve PyPI security. However, it needs to be done carefully. Certainly > Holger's idea of looking at how other programming language communities have > done it is a good one. > > A potential problem with the "no new package in six months" heuristic is > that it would punish mature packages with little or no improvements left. > Would one defeat this rule by simply uploading a "new" package every six > months? I think Tres was referring to the first release. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From maphew at gmail.com Sat Jun 1 08:33:51 2013 From: maphew at gmail.com (Matt Wilkie) Date: Fri, 31 May 2013 23:33:51 -0700 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: References: <51A76B55.5030207@cryptelium.net> <7F3F8244-8940-4D30-9FC2-236297581A55@stufft.io> <2AC2FE75-B60F-4AB9-92A1-F50001AC6039@stufft.io> Message-ID: sorry, here's the direct branch link : http://bazaar.launchpad.net/~leo-editor-team/leo-editor/trunk3/files and nightly snapshots: http://www.greygreen.org/leo/ -matt On Fri, May 31, 2013 at 11:30 PM, Matt Wilkie wrote: >> Matt, is your code available online anywhere? > > yes: https://code.launchpad.net/leo-editor > > thanks for looking at this. > > -matt From holger at merlinux.eu Sat Jun 1 08:48:07 2013 From: holger at merlinux.eu (holger krekel) Date: Sat, 1 Jun 2013 06:48:07 +0000 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> Message-ID: <20130601064807.GF7789@merlinux.eu> On Fri, May 31, 2013 at 15:18 +0200, Lennart Regebro wrote: > On Fri, May 31, 2013 at 3:09 PM, holger krekel wrote: > > I think there should be a PEP regulating the removal and the "taking over" > > process for packages. Your considerations make sense to me there. > > ASFAIK Perl has such policies a decade or so. Probably makes sense to > > use their provisions as a blue print. Such a PEP should contain a > > list of projects that are to be removed retro-actively if they don't > > comply within 6 months or so. I think the barrier shouldn't be too high, > > a valid email address and/or release activity are a good minimum. > > And it should be a short PEP :) > > I'd be OK with after six months automatically removing packages that > has only one owner/maintainer, and that owner/maintainer has no other > packages, and the package has no available downloads, and no contact > information on either package nor registered user. > > For other cases there should also be a manual way or removing packages > on the discretion of PyPI maintainers, as well as giving owner access > to packages whose owner is unreachable (I think there may already > exist a process for that). I'd like to argue in the same direction. While technical means to detect unmaintained or empty registrations are worthwhile to consdier i believe we should put emphasize on the human process. If you look at perl's guidelines for taking over maintenance here: http://www.cpan.org/misc/cpan-faq.html#How_adopt_module it's focused on the human communication process which i think is the right way to go about it. Any technical automated solution can probably be abused easily, given ill intentions, see the DNS for reference. A PEP should list criteria but leave decisions ultimately to a trusted board/admins which should maintain a public changelog of their actions. Those criteria should of course be automatically analyzed and used as input for the human process if possible. best, holger > //Lennart > From maphew at gmail.com Sat Jun 1 08:30:19 2013 From: maphew at gmail.com (Matt Wilkie) Date: Fri, 31 May 2013 23:30:19 -0700 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: <2AC2FE75-B60F-4AB9-92A1-F50001AC6039@stufft.io> References: <51A76B55.5030207@cryptelium.net> <7F3F8244-8940-4D30-9FC2-236297581A55@stufft.io> <2AC2FE75-B60F-4AB9-92A1-F50001AC6039@stufft.io> Message-ID: > Matt, is your code available online anywhere? yes: https://code.launchpad.net/leo-editor thanks for looking at this. -matt From donald at stufft.io Sat Jun 1 11:40:42 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 1 Jun 2013 05:40:42 -0400 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: References: <51A76B55.5030207@cryptelium.net> <7F3F8244-8940-4D30-9FC2-236297581A55@stufft.io> <2AC2FE75-B60F-4AB9-92A1-F50001AC6039@stufft.io> Message-ID: <619B8DAE-8716-430B-8053-D28A83CE4ED7@stufft.io> On Jun 1, 2013, at 2:33 AM, Matt Wilkie wrote: > sorry, here's the direct branch link : > http://bazaar.launchpad.net/~leo-editor-team/leo-editor/trunk3/files > > and nightly snapshots: http://www.greygreen.org/leo/ > > -matt > > On Fri, May 31, 2013 at 11:30 PM, Matt Wilkie wrote: >>> Matt, is your code available online anywhere? >> >> yes: https://code.launchpad.net/leo-editor >> >> thanks for looking at this. >> >> -matt As far as I can tell your issue is unrelated to the CDN. I don't know how to use Launchpad very well, but you hit a path in the code that, as far as I know, only gets hit when you don't have a long_description. This bit of code attempts to extract a long_description from a README file. It explicitly checks for certain extensions and TXT (instead of txt) is not one of them. However it returns a singular None when the calling code expects a 2 item tuple to be returned. I'm guessing that you didn't have a long_description when it wasn't working, and your README file was named README.TXT, and between when it wasn't working and when it was you added a long_description. Is my guess correct? The bit of code in question is https://bitbucket.org/pypa/pypi/src/0fd0fd89b791bc2641382f81dcc677ae4742abf7/description_utils.py?at=default#cl-202 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris at simplistix.co.uk Sat Jun 1 17:00:05 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 01 Jun 2013 16:00:05 +0100 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> Message-ID: <51AA0C75.6070805@simplistix.co.uk> On 31/05/2013 21:45, Noah Kantrowitz wrote: > > On May 31, 2013, at 1:34 PM, Tres Seaver wrote: > >> On 05/31/2013 09:18 AM, Lennart Regebro wrote: >>> I'd be OK with after six months automatically removing packages that >>> has only one owner/maintainer, and that owner/maintainer has no other >>> packages, and the package has no available downloads, and no contact >>> information on either package nor registered user. >> >> Why all the extras: if somebody wants to claim a project name, but can't >> upload a release for six months, they should just lose. I would actually >> be willing to have that cut down to a day: trying to grab the name >> before registering / uploading a release should result in loss of the claim. > > +1, I think this should just be treated as a form validation thing. It is a detail of the protocol that you upload a dist definition before the files, but I don't think we should consider it a valid PyPI entry until a file is uploaded (especially now that the default mode is to not scrape external sites). As we switch to not scraping, anything with no files should just vanish IMO, at which point it is available for registration again. If someone happens to ninja-upload between the setup.py register and setup.py upload, I think we can just throw an error message since chances of that happening are so amazingly low. I'm afraid I need to -1 on this. If I'm developing a new package, I try and avoid putting a release on PyPI until I have something stable, but I'll often put up an entry on PyPI explaining where the code is being developed. Working on a project for a year only to find that someone else has stolen your package name on PyPI (playing devils advocate here, lets say they saw the development of GitHub and knew a release was brewing, etc) and having to go through and rename everything seems unfairly painful... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Sat Jun 1 17:57:21 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 1 Jun 2013 11:57:21 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. Message-ID: In the Python community, we've been pretty laid back about how we name packages. When we were small, this made sense. It doesn't make sense any more. We should not have to come up with a process for recognizing squatters on simple package names. We should have something more systematic, IMO. Unfortunately, I think the sanest way of avoiding most package name issues is to base them on domains, as is done in the Java world. This goes against the Python philosophy of preferring flat to nested, but I still think it's better than trying to police squatters, or to encouraging races to claim top-level names. For a while, many of us have been pretty careful to use namespaces for new packages to mitigate this issue. For example, the zc namespace is a shorter version of com.zope, but at some point, it won't be fair for us to claim zc for ourselves. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From donald at stufft.io Sat Jun 1 20:01:26 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 1 Jun 2013 14:01:26 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: On Jun 1, 2013, at 11:57 AM, Jim Fulton wrote: > In the Python community, we've been pretty laid back > about how we name packages. When we were small, this made > sense. It doesn't make sense any more. > > We should not have to come up with a process for recognizing squatters > on simple package names. We should have something more systematic, > IMO. > > Unfortunately, I think the sanest way of avoiding most package > name issues is to base them on domains, as is done in the Java > world. This goes against the Python philosophy of preferring > flat to nested, but I still think it's better than trying to police squatters, > or to encouraging races to claim top-level names. > > For a while, many of us have been pretty careful to use namespaces > for new packages to mitigate this issue. For example, the zc namespace > is a shorter version of com.zope, but at some point, it won't be fair > for us to claim zc for ourselves. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I am opposed to this. Requiring someone to have purchased a domain adds a significant to publishing a project. If there are no requirements that they have purchased the domain then it's nothing more than a convention and something that anyone who wants to do this can do. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat Jun 1 20:02:08 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 1 Jun 2013 14:02:08 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: <1B273A3E-3085-42AC-967E-8C66C7A592F2@stufft.io> On Jun 1, 2013, at 2:01 PM, Donald Stufft wrote: > > I am opposed to this. Requiring someone to have purchased a domain adds a significant > to publishing a project. If there are no requirements that they have purchased the domain > then it's nothing more than a convention and something that anyone who wants to do > this can do. > Herp derp I can word good, obviously I mean a significant barrier to publishing. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jim at zope.com Sat Jun 1 20:09:21 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 1 Jun 2013 14:09:21 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <1B273A3E-3085-42AC-967E-8C66C7A592F2@stufft.io> References: <1B273A3E-3085-42AC-967E-8C66C7A592F2@stufft.io> Message-ID: On Sat, Jun 1, 2013 at 2:02 PM, Donald Stufft wrote: > > On Jun 1, 2013, at 2:01 PM, Donald Stufft wrote: > > > I am opposed to this. Requiring someone to have purchased a domain adds a > significant > to publishing a project. If there are no requirements that they have > purchased the domain > then it's nothing more than a convention and something that anyone who wants > to do > this can do. Fair enough. A common variation on this scenario, which avoids purchasing a domain, is to use a code hosting domain and project name, so, for example: org.bitbucket.j1m.foo. Of course, using a domain name without owning it is a form of squatting. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From noah at coderanger.net Sat Jun 1 20:54:57 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sat, 1 Jun 2013 11:54:57 -0700 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <1B273A3E-3085-42AC-967E-8C66C7A592F2@stufft.io> Message-ID: <142C33B9-63FE-4AC9-9BB1-FED5E66E5D39@coderanger.net> On Jun 1, 2013, at 11:09 AM, Jim Fulton wrote: > On Sat, Jun 1, 2013 at 2:02 PM, Donald Stufft wrote: >> >> On Jun 1, 2013, at 2:01 PM, Donald Stufft wrote: >> >> >> I am opposed to this. Requiring someone to have purchased a domain adds a >> significant >> to publishing a project. If there are no requirements that they have >> purchased the domain >> then it's nothing more than a convention and something that anyone who wants >> to do >> this can do. > > Fair enough. A common variation on this scenario, which avoids > purchasing a domain, > is to use a code hosting domain and project name, so, for example: > org.bitbucket.j1m.foo. > > Of course, using a domain name without owning it is a form of squatting. All that means is either we move the problem (instead of one shared namespace we two or three common ones) or we do it github-style and just prepend usernames at which point you can skip the whole URI thing because usernames must be unique for reasons of general sanity and I don't think it is a huge deal that a single person can't have two packages of the same name. Github-style namespacing just means that either names all suck (django/django, kennethreitz/requests) or you need to come up with some way to map un-namespaced names to their canonical form and we are more or less back at square one. If people don't mind the sucky names, they can already put that in their package name if the bare version is taken, so QED this is already doable in the current system, it just looks so ugly that no one wants to do it and enforcing the ugly seems like a poor option. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sat Jun 1 21:20:05 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 1 Jun 2013 20:20:05 +0100 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <51AA0C75.6070805@simplistix.co.uk> References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> Message-ID: On 1 June 2013 16:00, Chris Withers wrote: > I'm afraid I need to -1 on this. If I'm developing a new package, I try > and avoid putting a release on PyPI until I have something stable, but I'll > often put up an entry on PyPI explaining where the code is being developed. > > Working on a project for a year only to find that someone else has stolen > your package name on PyPI (playing devils advocate here, lets say they saw > the development of GitHub and knew a release was brewing, etc) and having > to go through and rename everything seems unfairly painful... I'd say that at a minimum, no deletion should happen without the author being contacted. As long as there's an author email, and the author actually replies to emails saying "do you mind if I take over this project name". I'm -1 on anything that doesn't involve at least a minimal level of human involvement (possibly excepting an initial clean up exercise for projects with no author email) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Sat Jun 1 21:21:12 2013 From: holger at merlinux.eu (holger krekel) Date: Sat, 1 Jun 2013 19:21:12 +0000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: <20130601192112.GG7789@merlinux.eu> On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: > In the Python community, we've been pretty laid back > about how we name packages. When we were small, this made > sense. It doesn't make sense any more. I've heart this sentiment before, but would like to read more clearly stated problems. > We should not have to come up with a process for recognizing squatters > on simple package names. We should have something more systematic, > IMO. > > Unfortunately, I think the sanest way of avoiding most package > name issues is to base them on domains, as is done in the Java > world. This goes against the Python philosophy of preferring > flat to nested, but I still think it's better than trying to police squatters, > or to encouraging races to claim top-level names. I am not sure that tying to DNS namespacing is the only solution here (whatever the problem is exactly :). > For a while, many of us have been pretty careful to use namespaces > for new packages to mitigate this issue. For example, the zc namespace > is a shorter version of com.zope, but at some point, it won't be fair > for us to claim zc for ourselves. I wonder if we could allow people/groups to apply (to humans) for a namespace which they can subsequently control, like the "zc.*" one. Everyone could continue to push non-namespaced (flat) packages to pypi like now but the names couldn't take the form of namespaced ones. So for example if the django community wants to introduce the concept of "vetted" plugins/addons, they could move to manage "dj.*" or so. I don't think we would suddenly drown in namespace regs if we make it a pre-condition that there need to be a couple of existing real packages that would go into it. cheers, holger > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From donald at stufft.io Sat Jun 1 21:30:15 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 1 Jun 2013 15:30:15 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <20130601192112.GG7789@merlinux.eu> References: <20130601192112.GG7789@merlinux.eu> Message-ID: On Jun 1, 2013, at 3:21 PM, holger krekel wrote: > On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: >> In the Python community, we've been pretty laid back >> about how we name packages. When we were small, this made >> sense. It doesn't make sense any more. > > I've heart this sentiment before, but would like to read more > clearly stated problems. > >> We should not have to come up with a process for recognizing squatters >> on simple package names. We should have something more systematic, >> IMO. >> >> Unfortunately, I think the sanest way of avoiding most package >> name issues is to base them on domains, as is done in the Java >> world. This goes against the Python philosophy of preferring >> flat to nested, but I still think it's better than trying to police squatters, >> or to encouraging races to claim top-level names. > > I am not sure that tying to DNS namespacing is the only solution here > (whatever the problem is exactly :). > >> For a while, many of us have been pretty careful to use namespaces >> for new packages to mitigate this issue. For example, the zc namespace >> is a shorter version of com.zope, but at some point, it won't be fair >> for us to claim zc for ourselves. > > I wonder if we could allow people/groups to apply (to humans) for a > namespace which they can subsequently control, like the "zc.*" one. > > Everyone could continue to push non-namespaced (flat) packages to pypi > like now but the names couldn't take the form of namespaced ones. > > So for example if the django community wants to introduce the concept > of "vetted" plugins/addons, they could move to manage "dj.*" or so. > > I don't think we would suddenly drown in namespace regs if we make > it a pre-condition that there need to be a couple of existing real > packages that would go into it. I've long thought about allowing people to claim a namespace. It certainly provides a "carrot" to get people to namespace their packages and thus not take up the top level names. > > cheers, > holger > >> Jim >> >> -- >> Jim Fulton >> http://www.linkedin.com/in/jimfulton >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Sat Jun 1 22:29:09 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 1 Jun 2013 22:29:09 +0200 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> Message-ID: On Sat, Jun 1, 2013 at 9:20 PM, Paul Moore wrote: > I'm -1 on anything that doesn't involve at least a minimal level of human > involvement (possibly excepting an initial clean up exercise for projects > with no author email) This is why I basically said I'm OK with automatic deletion after a time if there are no downloadable packages and no contact information. Otherwise the owner should be contacted. From jaraco at jaraco.com Sat Jun 1 22:13:27 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sat, 1 Jun 2013 20:13:27 +0000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130601192112.GG7789@merlinux.eu> Message-ID: <10b206d3b6e0451ebb85355ef0c8ea04@BY2PR06MB011.namprd06.prod.outlook.com> From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Donald Stufft Sent: Saturday, 01 June, 2013 15:30 To: holger krekel Cc: distutils sig Subject: Re: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. On Jun 1, 2013, at 3:21 PM, holger krekel > wrote: On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: For a while, many of us have been pretty careful to use namespaces for new packages to mitigate this issue. For example, the zc namespace is a shorter version of com.zope, but at some point, it won't be fair for us to claim zc for ourselves. I wonder if we could allow people/groups to apply (to humans) for a namespace which they can subsequently control, like the "zc.*" one. So for example if the django community wants to introduce the concept of "vetted" plugins/addons, they could move to manage "dj.*" or so. I think this example highlights some of the challenges with registering/controlling namespaces - who owns what and what is the meaning of a (distribution) package name? For example, what is the namespace used for an endorsed django plugin written by zope corporation? This problem is not present now, as the author can choose the domain which is most relevant to that plugin and its users. If there's some expectation that it should appear in a namespace managed by another organization, that necessitates a coordination between the namespace owner/manager and the project author. I think more people would claim namespaces when namespaces are better supported in Python. My expectation is Python 3.3 namespace package support will ease that challenge (when it becomes a dominant version). My inclination is to say it's not a huge problem, and later is preferable than sooner. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From ralf at systemexit.de Sat Jun 1 23:26:14 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Sat, 01 Jun 2013 23:26:14 +0200 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server In-Reply-To: <20130530081936.GP7789@merlinux.eu> (holger krekel's message of "Thu, 30 May 2013 08:19:36 +0000") References: <87ip227k9y.fsf@myhost.lan> <51A62162.1000901@simplistix.co.uk> <20130530081936.GP7789@merlinux.eu> Message-ID: <87k3mdsetl.fsf@myhost.lan> holger krekel writes: >> How does it compare to http://pypi.python.org/pypi/devpi-server? > > the main differences as i see them (note i am the devpi-server author, though): > > - pypiserver supports uploading to private indexes, > devpi-server not yet (but next week / on trunk already :) I still consider that a bug. scp works much better. I'm not sure if devpi-server maintains some kind of database, but I consider it a big plus to be able to just copy files and directories to the package directory and serve them instantly. > > - pypiserver has no dependencies (ships bottle inline), > devpi-server depends on redis (to be dropped next week for no-dep fs-storage) > and bottle, requests, py, all proven libraries. nice, I'll try it if the redis dependency is gone. > > - pypiserver redirects the lookup of pypi packages to pypi.python.org, that's related to a weak point of pypiserver: you currently have to parse the log files and look for HTTP redirects if you like to know what packages are missing in the local package directory. you also can't install a specific version of a package if the package directory doesn't have that specific version but a different version. this may or may not be good depending on your use case. redirects can also be completely disabled, so pip/easy_install only see the packages in the package directory. > devpi-server caches them and serves everything (including #egg-links) > through itself, allowing complete offline operations (including > caching/serving of 3rd party site's packages) using the > extended PEP381 mirror protocol > > - pypiserver has a good and simple implementation, devpi-server is > little more complex mostly due to its caching/crawling logic. > > - both are very well tested and maintained but pypiserver is out there > for a longer time already, so has seen more RL-testing. > > Ralph, please add/comment as you see fit. pypiserver exhibits it's WSGI application and might be easier to deploy using different WSGI servers. -- Cheers Ralf From p.f.moore at gmail.com Sun Jun 2 00:00:23 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 1 Jun 2013 23:00:23 +0100 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: On 1 June 2013 16:57, Jim Fulton wrote: > In the Python community, we've been pretty laid back > about how we name packages. When we were small, this made > sense. It doesn't make sense any more. I'd like to see some evidence that this is the case. It doesn't seem so to me - most package names are relatively discoverable and/or intuitive, and we currently have basically no namespacing. There's a long way to go before something like your suggestion is needed, in my view. Unfortunately, I think the sanest way of avoiding most package > name issues is to base them on domains, as is done in the Java > world. This goes against the Python philosophy of preferring > flat to nested, but I still think it's better than trying to police > squatters, > or to encouraging races to claim top-level names. > No, no, no... There's the point Donald made that you require people to own a domain (or you create some sort of hack like org.bitbucket.username/com.github.username/...) but it also makes package names unreasonably deep, and requires an explosion of namespace packages at the top level. And it's ugly :-) Perl manages with a relatively flat namespace and relatively informal rules for managing package names (AIUI). I'm sure Python can, too. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sun Jun 2 00:35:42 2013 From: dholth at gmail.com (Daniel Holth) Date: Sat, 1 Jun 2013 18:35:42 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: On Sat, Jun 1, 2013 at 6:00 PM, Paul Moore wrote: > > On 1 June 2013 16:57, Jim Fulton wrote: >> In the Python community, we've been pretty laid back >> about how we name packages. When we were small, this made >> sense. It doesn't make sense any more. > > I'd like to see some evidence that this is the case. It doesn't seem so to > me - most package names are relatively discoverable and/or intuitive, and we > currently have basically no namespacing. There's a long way to go before > something like your suggestion is needed, in my view. > >> Unfortunately, I think the sanest way of avoiding most package >> name issues is to base them on domains, as is done in the Java >> world. This goes against the Python philosophy of preferring >> flat to nested, but I still think it's better than trying to police >> squatters, >> or to encouraging races to claim top-level names. > > > No, no, no... > > There's the point Donald made that you require people to own a domain (or > you create some sort of hack like > org.bitbucket.username/com.github.username/...) but it also makes package > names unreasonably deep, and requires an explosion of namespace packages at > the top level. And it's ugly :-) > > Perl manages with a relatively flat namespace and relatively informal rules > for managing package names (AIUI). I'm sure Python can, too. > > Paul There's also the fact that our module namespace is separate from our distribution names namespace... From ncoghlan at gmail.com Sun Jun 2 01:32:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 2 Jun 2013 09:32:05 +1000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: I'm with Jason in the "maybe eventually, but not right now" camp. Namespace collisions are indeed a possibility and a potential concern, both in the distribution namespace and the top level import namespace. The fact there is no 1to1 mapping between distribution names and the import namespace means that informal conflict avoidance is already possible - prepending "-" to the desired package name makes it possible to publish it alongside another distribution using the same name without having to change the top level import location. If the distributed packages use explicit relative imports appropriately, an integrator may even be able to use them side by side by dropping them into higher level namespace packages. Java's "use the domain name" approach simply outsources the conflict resolution to a third party, by *requiring* that publishers acquire a domain name prior to publication. I prefer our model of initially *assuming* a lack of conflict to lower barriers to publication. I do think we need to better handle cases where the assumption breaks down, but we shouldn't forget namespacing is already possible. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Sun Jun 2 02:34:53 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 01 Jun 2013 20:34:53 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51A90BD6.5080801@students.poly.edu> <51A92525.50307@students.poly.edu> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/31/2013 06:41 PM, Jim Fulton wrote: > I hope no one is proposing removing a project simply because someone > hasn't released a new (after initial) release in some period of time. Certainly not I: I'd just like to prevent "squatters" (no matter how well-intentioned, Withers!) from tying up a project name indefinitely. Code talks, as it were: release something or take your marbles and go home. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGqky0ACgkQ+gerLs4ltQ74iwCgqSr4TqKC/ktgU0XlsDKedp2i 3ZwAnRnuuEX2imZlVs2CT54faUZFYo6l =qReS -----END PGP SIGNATURE----- From holger at merlinux.eu Sun Jun 2 09:10:18 2013 From: holger at merlinux.eu (holger krekel) Date: Sun, 2 Jun 2013 07:10:18 +0000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: <20130602071018.GH7789@merlinux.eu> On Sun, Jun 02, 2013 at 09:32 +1000, Nick Coghlan wrote: > I'm with Jason in the "maybe eventually, but not right now" camp. > > Namespace collisions are indeed a possibility and a potential concern, both > in the distribution namespace and the top level import namespace. Indeed, in this thread, i assumed we were only talking about distribution/pypi namespacing. Somewhat proper import namespacing is only available with very recent python versions which still have a long way to become mainstream. > The fact there is no 1to1 mapping between distribution names and the import > namespace means that informal conflict avoidance is already possible - > prepending "-" to the desired package name makes it possible to > publish it alongside another distribution using the same name without > having to change the top level import location. If the distributed packages > use explicit relative imports appropriately, an integrator may even be able > to use them side by side by dropping them into higher level namespace > packages. > > Java's "use the domain name" approach simply outsources the conflict > resolution to a third party, by *requiring* that publishers acquire a > domain name prior to publication. I prefer our model of initially > *assuming* a lack of conflict to lower barriers to publication. > > I do think we need to better handle cases where the assumption breaks down, > but we shouldn't forget namespacing is already possible. If pypi has no idea about namespaces (like i considered them in my other post) then using namespaces do not really provide much. Someone can still come along and publish within that pseudo-namespace. I would think the goal of pypi-namespaces would be to give a group control over anything that's released using it, allowing to communicate install-users certain guarantees. However, before further discussion i think there first needs to be more reasoning and stating of practical problems with the current anyone-can-register-anything-that's-not-taken model. best, holger > Cheers, > Nick. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Sun Jun 2 09:26:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 2 Jun 2013 17:26:07 +1000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <20130602071018.GH7789@merlinux.eu> References: <20130602071018.GH7789@merlinux.eu> Message-ID: On Sun, Jun 2, 2013 at 5:10 PM, holger krekel wrote: > If pypi has no idea about namespaces (like i considered them in my other post) > then using namespaces do not really provide much. Someone can still come along > and publish within that pseudo-namespace. I would think the goal of > pypi-namespaces would be to give a group control over anything that's > released using it, allowing to communicate install-users certain guarantees. > > However, before further discussion i think there first needs to be more > reasoning and stating of practical problems with the current > anyone-can-register-anything-that's-not-taken model. TUF actually has native support for prefix delegation, but actually *using* that is a long way down the todo list at the moment. Static dependency metadata publication and end-to-end signature support are well ahead of it and will likely keep us collectively busy for a while yet. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Sun Jun 2 09:34:30 2013 From: holger at merlinux.eu (holger krekel) Date: Sun, 2 Jun 2013 07:34:30 +0000 Subject: [Distutils] [ANN] pypiserver 1.1.1 - minimal private pypi server In-Reply-To: <87k3mdsetl.fsf@myhost.lan> References: <87ip227k9y.fsf@myhost.lan> <51A62162.1000901@simplistix.co.uk> <20130530081936.GP7789@merlinux.eu> <87k3mdsetl.fsf@myhost.lan> Message-ID: <20130602073430.GI7789@merlinux.eu> On Sat, Jun 01, 2013 at 23:26 +0200, Ralf Schmitt wrote: > holger krekel writes: > > >> How does it compare to http://pypi.python.org/pypi/devpi-server? > > > > the main differences as i see them (note i am the devpi-server author, though): > > > > - pypiserver supports uploading to private indexes, > > devpi-server not yet (but next week / on trunk already :) > > I still consider that a bug. scp works much better. I'm not sure if > devpi-server maintains some kind of database, but I consider it a big > plus to be able to just copy files and directories to the package > directory and serve them instantly. (devpi-server maintains a database, indeed). If you scp a large file and someone else is about the pip-install that package, could you get partial downloads which fail on the user side? I like the ability of transfering things via scp but i guess in devpi-server i'd make them go to a "clearing area" where they are screened before put up for serving. > ... > > - pypiserver redirects the lookup of pypi packages to pypi.python.org, > > that's related to a weak point of pypiserver: you currently have to > parse the log files and look for HTTP redirects if you like to know what > packages are missing in the local package directory. It's a recursive problem because those packages might have pulled in yet more dependencies. > you also can't install a specific version of a package if the package > directory doesn't have that specific version but a different version. > this may or may not be good depending on your use case. > > redirects can also be completely disabled, so pip/easy_install only see > the packages in the package directory. FYI devpi-server supports multiple indexes and you can define inheritance semantics between indexes. If an index A inherits from another index B (which can be the full pypi-mirroring index) then queries on A will see all release files from index B merged in. By default, there also is a "root/prod" index which does not inherit anything and contains only the packages explicitely pushed to it. If you install from such a no-bases index, only its own list of release files are considered. This is designed such that users can start off with "play areas" to prepare and test their packages including third party ones before "staging" them to a more constrained no-bases index. > > devpi-server caches them and serves everything (including #egg-links) > > through itself, allowing complete offline operations (including > > caching/serving of 3rd party site's packages) using the > > extended PEP381 mirror protocol > > > > - pypiserver has a good and simple implementation, devpi-server is > > little more complex mostly due to its caching/crawling logic. > > > > - both are very well tested and maintained but pypiserver is out there > > for a longer time already, so has seen more RL-testing. > > > > Ralph, please add/comment as you see fit. > > pypiserver exhibits it's WSGI application and might be easier to deploy > using different WSGI servers. In principle, A wsgi-app is also available with devpi-server but there is one issue i am unsure about: the wsgi-app needs to have async threads/greenlets running to keep the pypi-mirror cache up-to-date. Starting those threads at wsgi-app creation seems too early so it currently starts them before bottle.run() is invoked. I guess it could also happen at "first-request" time or so which should make the creation of the WSGI-app become a no-sideeffect operation. cheers, holger > -- > Cheers > Ralf > From holger at merlinux.eu Sun Jun 2 09:37:49 2013 From: holger at merlinux.eu (holger krekel) Date: Sun, 2 Jun 2013 07:37:49 +0000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130602071018.GH7789@merlinux.eu> Message-ID: <20130602073749.GJ7789@merlinux.eu> On Sun, Jun 02, 2013 at 17:26 +1000, Nick Coghlan wrote: > On Sun, Jun 2, 2013 at 5:10 PM, holger krekel wrote: > > If pypi has no idea about namespaces (like i considered them in my other post) > > then using namespaces do not really provide much. Someone can still come along > > and publish within that pseudo-namespace. I would think the goal of > > pypi-namespaces would be to give a group control over anything that's > > released using it, allowing to communicate install-users certain guarantees. > > > > However, before further discussion i think there first needs to be more > > reasoning and stating of practical problems with the current > > anyone-can-register-anything-that's-not-taken model. > > TUF actually has native support for prefix delegation, but actually > *using* that is a long way down the todo list at the moment. Static > dependency metadata publication and end-to-end signature support are > well ahead of it and will likely keep us collectively busy for a while > yet. No worries, I understood already that it's not high on your list. I'd appreciate, however, if Jim or someone else could state the problems with missing namespacing so we can start a discussion from there later. Speaking of TUF: is there some kind of PEP like doc floating already? cheers, holger > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > From ncoghlan at gmail.com Sun Jun 2 10:21:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 2 Jun 2013 18:21:05 +1000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <20130602073749.GJ7789@merlinux.eu> References: <20130602071018.GH7789@merlinux.eu> <20130602073749.GJ7789@merlinux.eu> Message-ID: On Sun, Jun 2, 2013 at 5:37 PM, holger krekel wrote: > Speaking of TUF: is there some kind of PEP like doc floating already? Just the proof-of-concept the TUF folks created about using it to secure /simple. I'm personally sold on the technology itself as something we should deploy in the long run, but I think it makes sense to wait until we have the static dependency metadata publication and various other PyPI related infrastructure issues sorted out before we try to offer additional protection above and beyond trusting the SSL CA system and PyPI itself. That said, one of the reasons PEP 426 calls out the "essential dependency resolution" fields is that those are the ones I think it may make sense to embed in the TUF custom metadata fields. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From martin at v.loewis.de Sun Jun 2 12:51:04 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 02 Jun 2013 12:51:04 +0200 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> Message-ID: <20130602125104.Horde.ymzUqA32YbvcCiM0TU_gzg1@webmail.df.eu> Quoting Paul Moore : > I'm -1 on anything that doesn't involve at least a minimal level of human > involvement (possibly excepting an initial clean up exercise for projects > with no author email) I support this position. This is actually how PyPI has operated over the last decade. People have always taken over projects, either the project entirely, or just the name. It always involved contacting the original owner of the name. In this thread, Lukas wrote > Fortunately we were able to work it out with Richard > but we had to contact him directly and waste his cycles on this. I don't consider his cycles wasted at all. It's an important interaction. I'm fine with formalizing the process, and I'm also fine with adding tool support. However, I agree that a PEP should be written and agreed about this. Personally, I'd favor this procedure: - nothing happens unless some user explicitly requests it - on request, the owner is contacted, and given some time to respond - if they do respond, and are unwilling to yield the name, nothing happens - if they have confirmed that they want to keep the name, they won't be asked again for at least one year. Regards, Martin From donald at stufft.io Sun Jun 2 13:54:52 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 2 Jun 2013 07:54:52 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <20130602125104.Horde.ymzUqA32YbvcCiM0TU_gzg1@webmail.df.eu> References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> <20130602125104.Horde.ymzUqA32YbvcCiM0TU_gzg1@webmail.df.eu> Message-ID: On Jun 2, 2013, at 6:51 AM, martin at v.loewis.de wrote: > > Quoting Paul Moore : > >> I'm -1 on anything that doesn't involve at least a minimal level of human >> involvement (possibly excepting an initial clean up exercise for projects >> with no author email) > > I support this position. This is actually how PyPI has operated over the last > decade. People have always taken over projects, either the project entirely, > or just the name. It always involved contacting the original owner of the name. > > In this thread, Lukas wrote > >> Fortunately we were able to work it out with Richard >> but we had to contact him directly and waste his cycles on this. > > I don't consider his cycles wasted at all. It's an important interaction. > > I'm fine with formalizing the process, and I'm also fine with adding tool > support. However, I agree that a PEP should be written and agreed about this. > > Personally, I'd favor this procedure: > - nothing happens unless some user explicitly requests it > - on request, the owner is contacted, and given some time to respond > - if they do respond, and are unwilling to yield the name, nothing > happens > - if they have confirmed that they want to keep the name, they won't > be asked again for at least one year. The missing case here is what happens if they don't respond? > > Regards, > Martin > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Jun 2 14:09:20 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 2 Jun 2013 08:09:20 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130602071018.GH7789@merlinux.eu> <20130602073749.GJ7789@merlinux.eu> Message-ID: On Jun 2, 2013, at 4:21 AM, Nick Coghlan wrote: > On Sun, Jun 2, 2013 at 5:37 PM, holger krekel wrote: >> Speaking of TUF: is there some kind of PEP like doc floating already? > > Just the proof-of-concept the TUF folks created about using it to > secure /simple. I'm personally sold on the technology itself as > something we should deploy in the long run, but I think it makes sense > to wait until we have the static dependency metadata publication and > various other PyPI related infrastructure issues sorted out before we > try to offer additional protection above and beyond trusting the SSL > CA system and PyPI itself. > > That said, one of the reasons PEP 426 calls out the "essential > dependency resolution" fields is that those are the ones I think it > may make sense to embed in the TUF custom metadata fields. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig If we deploy some sort of end to end signing I think TUF is a good implementation of it. I'm not sold on the possibility of reasonably doing end to end signing here though. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun Jun 2 15:01:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 2 Jun 2013 23:01:22 +1000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130602071018.GH7789@merlinux.eu> <20130602073749.GJ7789@merlinux.eu> Message-ID: On Sun, Jun 2, 2013 at 10:09 PM, Donald Stufft wrote: > If we deploy some sort of end to end signing I think TUF is a good > implementation of it. > > I'm not sold on the possibility of reasonably doing end to end signing here > though. I think in the long run it's a technology we want to offer, but even with it deployed PyPI would continue to act as a trusted intermediary in most cases. Effective key management is such a PITA that only a few larger projects would be in a real position to take direct advantage of end-to-end signing - for the remaining projects, trusting PyPI not to get compromised is already the status quo. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tk47 at students.poly.edu Sun Jun 2 15:42:49 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Sun, 2 Jun 2013 09:42:49 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130602071018.GH7789@merlinux.eu> <20130602073749.GJ7789@merlinux.eu> Message-ID: <51AB4BD9.4040303@students.poly.edu> On 6/2/13 4:21 AM, Nick Coghlan wrote: > On Sun, Jun 2, 2013 at 5:37 PM, holger krekel wrote: >> Speaking of TUF: is there some kind of PEP like doc floating already? > > Just the proof-of-concept the TUF folks created about using it to > secure /simple. I'm personally sold on the technology itself as > something we should deploy in the long run, but I think it makes sense > to wait until we have the static dependency metadata publication and > various other PyPI related infrastructure issues sorted out before we > try to offer additional protection above and beyond trusting the SSL > CA system and PyPI itself. > > That said, one of the reasons PEP 426 calls out the "essential > dependency resolution" fields is that those are the ones I think it > may make sense to embed in the TUF custom metadata fields. > Nick got our proof-of-concept pretty much right, and I just want to make this correction: we offered security for both /simple and /packages, but only for a subset of packages. We were working on securing all the packages under PyPI, but were derailed by some projects with immediate deadlines. The good news is that we will be continuing our work full-time this summer, and expect to make much progress. We don't have a PEP for it, besides our design proposal[1]. I think a PEP is a good idea, and we should draft one along the way. [1] https://docs.google.com/document/d/1sHMhgrGXNCvBZdmjVJzuoN5uMaUAUDWBmn3jo7vxjjw/edit?usp=sharing From tk47 at students.poly.edu Sun Jun 2 15:52:40 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Sun, 2 Jun 2013 09:52:40 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130602071018.GH7789@merlinux.eu> <20130602073749.GJ7789@merlinux.eu> Message-ID: <51AB4E28.5080205@students.poly.edu> On 6/2/13 9:01 AM, Nick Coghlan wrote: > On Sun, Jun 2, 2013 at 10:09 PM, Donald Stufft wrote: >> If we deploy some sort of end to end signing I think TUF is a good >> implementation of it. >> >> I'm not sold on the possibility of reasonably doing end to end signing here >> though. > > I think in the long run it's a technology we want to offer, but even > with it deployed PyPI would continue to act as a trusted intermediary > in most cases. Effective key management is such a PITA that only a few > larger projects would be in a real position to take direct advantage > of end-to-end signing - for the remaining projects, trusting PyPI not > to get compromised is already the status quo. > Yes, key management could be a real PITA if we do not consider usability. In our design proposal, we talked about how to try to maximize usability and security, by keeping the truly critical keys offline (which would be used rarely), and the not-so-critical keys online (which means that automation can easily use them). We will be working on TUF and PyPI full-time this summer. As I write this, we are introducing additional security mechanisms for some cases which arise frequently; e.g. how do we tell TUF to put more trust in packages from a stable-release role versus a bleeding-edge role? From jim at zope.com Sun Jun 2 17:14:28 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 2 Jun 2013 11:14:28 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <142C33B9-63FE-4AC9-9BB1-FED5E66E5D39@coderanger.net> References: <1B273A3E-3085-42AC-967E-8C66C7A592F2@stufft.io> <142C33B9-63FE-4AC9-9BB1-FED5E66E5D39@coderanger.net> Message-ID: On Sat, Jun 1, 2013 at 2:54 PM, Noah Kantrowitz wrote: > > On Jun 1, 2013, at 11:09 AM, Jim Fulton wrote: > >> On Sat, Jun 1, 2013 at 2:02 PM, Donald Stufft wrote: >>> >>> On Jun 1, 2013, at 2:01 PM, Donald Stufft wrote: >>> >>> >>> I am opposed to this. Requiring someone to have purchased a domain adds a >>> significant >>> to publishing a project. If there are no requirements that they have >>> purchased the domain >>> then it's nothing more than a convention and something that anyone who wants >>> to do >>> this can do. >> >> Fair enough. A common variation on this scenario, which avoids >> purchasing a domain, >> is to use a code hosting domain and project name, so, for example: >> org.bitbucket.j1m.foo. >> >> Of course, using a domain name without owning it is a form of squatting. > > All that means is either we move the problem (instead of one shared namespace we two or three common ones) I don't understand why you say two or three. There would be as many namespaces as there are domains or VCS accounts. There would be many distinct namespaces, each controlled controlled by a single user or organization. > or we do it github-style and just prepend usernames at which point you can skip the whole URI thing because usernames must be unique for reasons of general sanity and I don't think it is a huge deal that a single person can't have two packages of the same name. That's an option. (I assume you mean PyPI user names.) It would be more attractive if PyPI supported organizational accounts. (I sure wish it did.) I can't say I find the idea of tying a package name to an account name attractive, but it's a good alternative for projects without a domain. > Github-style namespacing just means that either names all suck (django/django, kennethreitz/requests) or you need to come up with some way to map un-namespaced names to their canonical form and we are more or less back at square one. If people don't mind the sucky names, they can already put that in their package name if the bare version is taken, so QED this is already doable in the current system, it just looks so ugly that no one wants to do it and enforcing the ugly seems like a poor option. My observation of the java world is that most packages that get published to central repositories end up having domain based names. Even that sucks to some degree because flat is better than nested. I just don't think the current ad-hoc mechanism we're using now is scalable. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Sun Jun 2 17:25:22 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 2 Jun 2013 11:25:22 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <20130601192112.GG7789@merlinux.eu> References: <20130601192112.GG7789@merlinux.eu> Message-ID: On Sat, Jun 1, 2013 at 3:21 PM, holger krekel wrote: > On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: >> In the Python community, we've been pretty laid back >> about how we name packages. When we were small, this made >> sense. It doesn't make sense any more. > > I've heart this sentiment before, but would like to read more > clearly stated problems. I thought the problem was pretty clear: name collisions. There's a parallel thread on how to detect and reclaim names that are taken and unused. I think if we had a more systematic way of naming packages, this wouldn't be an issue. >> We should not have to come up with a process for recognizing squatters >> on simple package names. We should have something more systematic, >> IMO. >> >> Unfortunately, I think the sanest way of avoiding most package >> name issues is to base them on domains, as is done in the Java >> world. This goes against the Python philosophy of preferring >> flat to nested, but I still think it's better than trying to police squatters, >> or to encouraging races to claim top-level names. > > I am not sure that tying to DNS namespacing is the only solution here > (whatever the problem is exactly :). It's not the only solution, but it's a pretty easy one. It avoids (more more precisely reuses) a central naming authority. It's the technique used by the java ecosystem and by XML namespaces, for example. > >> For a while, many of us have been pretty careful to use namespaces >> for new packages to mitigate this issue. For example, the zc namespace >> is a shorter version of com.zope, but at some point, it won't be fair >> for us to claim zc for ourselves. > > I wonder if we could allow people/groups to apply (to humans) for a > namespace which they can subsequently control, like the "zc.*" one. We could. Maybe that's the most palatable alternative. It has the huge benefit of promoting relatively flat, Pythonic, namespaces. It has disadvantages: - We have to set up some sort of naming authority. - It's probably not scalable. > Everyone could continue to push non-namespaced (flat) packages to pypi > like now but the names couldn't take the form of namespaced ones. I'm not sure what you're suggesting here. Are you saying someone could publish a package named: "zc", bit not "zc.foo"? Or are you saying that publication of a package named "bar" would prevent someone from creating a "bar" namespace, and the other way around? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Sun Jun 2 17:35:07 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 2 Jun 2013 11:35:07 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <10b206d3b6e0451ebb85355ef0c8ea04@BY2PR06MB011.namprd06.prod.outlook.com> References: <20130601192112.GG7789@merlinux.eu> <10b206d3b6e0451ebb85355ef0c8ea04@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: On Sat, Jun 1, 2013 at 4:13 PM, Jason R. Coombs wrote: > > > > > From: Distutils-SIG > [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of > Donald Stufft > Sent: Saturday, 01 June, 2013 15:30 > To: holger krekel > Cc: distutils sig > Subject: Re: [Distutils] Sooner or later, we're going to have to be more > formal about how we name packages. > > > > > > On Jun 1, 2013, at 3:21 PM, holger krekel wrote: > > > > On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: > > > For a while, many of us have been pretty careful to use namespaces > for new packages to mitigate this issue. For example, the zc namespace > is a shorter version of com.zope, but at some point, it won't be fair > for us to claim zc for ourselves. > > > I wonder if we could allow people/groups to apply (to humans) for a > namespace which they can subsequently control, like the "zc.*" one. > > > > So for example if the django community wants to introduce the concept > > > of "vetted" plugins/addons, they could move to manage "dj.*" or so. > > > > I think this example highlights some of the challenges with > registering/controlling namespaces ? who owns what and what is the meaning > of a (distribution) package name? For example, what is the namespace used > for an endorsed django plugin written by zope corporation? IMO the purpose of the namespace is to organize names. Whoever owns the namespace decides what names can go into it. It's purely a name management issue, not, for example, a intellectual property issue. If Zope Corporation independently creates a Django plugin, I'd expect it to go into the "zc" namespace. OTOH, Zope Corporation was an active member of the Django community, it might publish to the "dj" (or whatever) namespace, or request permission to do so. I started using the zc namespace a few years ago because I didn't want to impose our names on the Zope community. > I think more people would claim namespaces when namespaces are better > supported in Python. My expectation is Python 3.3 namespace package support > will ease that challenge (when it becomes a dominant version). This is somewhat baffling to me. We've used namespaces for over a decade virtually without issue. (We;ve used namespaces far longer than that, going all the way back to "ni" with relatively minor issues.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Sun Jun 2 17:45:03 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 2 Jun 2013 11:45:03 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: On Sat, Jun 1, 2013 at 6:00 PM, Paul Moore wrote: > > On 1 June 2013 16:57, Jim Fulton wrote: >> In the Python community, we've been pretty laid back >> about how we name packages. When we were small, this made >> sense. It doesn't make sense any more. > > I'd like to see some evidence that this is the case. How about the "A process for removal of PyPi entries" thread? > It doesn't seem so to > me - most package names are relatively discoverable and/or intuitive, Um, boto? fabric? paramiko? kazoo? The way I find packages is by searching. Names are irrelevant. > and we > currently have basically no namespacing. We have a ton of namespacing. It's just informal. IMO, it's a result of good hygiene and citizenship (I don't mean to dis anyone not using namespaces). > There's a long way to go before > something like your suggestion is needed, in my view. If the projects now using namespaces weren't, I predict the problem would be a lot more apparent. > >> Unfortunately, I think the sanest way of avoiding most package >> name issues is to base them on domains, as is done in the Java >> world. This goes against the Python philosophy of preferring >> flat to nested, but I still think it's better than trying to police >> squatters, >> or to encouraging races to claim top-level names. > > > No, no, no... > > There's the point Donald made that you require people to own a domain (or > you create some sort of hack like > org.bitbucket.username/com.github.username/...) but it also makes package > names unreasonably deep, and requires an explosion of namespace packages at > the top level. And it's ugly :-) > > Perl manages with a relatively flat namespace and relatively informal rules > for managing package names (AIUI). I'm sure Python can, too. Perl's a dead language. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From donald at stufft.io Sun Jun 2 17:49:34 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 2 Jun 2013 11:49:34 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130601192112.GG7789@merlinux.eu> Message-ID: On Jun 2, 2013, at 11:25 AM, Jim Fulton wrote: > On Sat, Jun 1, 2013 at 3:21 PM, holger krekel wrote: >> On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: >>> In the Python community, we've been pretty laid back >>> about how we name packages. When we were small, this made >>> sense. It doesn't make sense any more. >> >> I've heart this sentiment before, but would like to read more >> clearly stated problems. > > I thought the problem was pretty clear: name collisions. > > There's a parallel thread on how to detect and reclaim > names that are taken and unused. I think if we had a more > systematic way of naming packages, this wouldn't be an issue. I think the parallel thread mostly involves people wanting specific names that are no longer available. I know I've claimed a few good names at the start of an idea then abandoned the idea and forgotten to deregister the name. There are still lots of good names but people get attached to ones they like and it feels kinda shitty when you have to rename your project because of a placeholder that's been there for 3 years that you didn't notice. > >>> We should not have to come up with a process for recognizing squatters >>> on simple package names. We should have something more systematic, >>> IMO. >>> >>> Unfortunately, I think the sanest way of avoiding most package >>> name issues is to base them on domains, as is done in the Java >>> world. This goes against the Python philosophy of preferring >>> flat to nested, but I still think it's better than trying to police squatters, >>> or to encouraging races to claim top-level names. >> >> I am not sure that tying to DNS namespacing is the only solution here >> (whatever the problem is exactly :). > > It's not the only solution, but it's a pretty easy one. It avoids (more > more precisely reuses) a central naming authority. It's the technique > used by the java ecosystem and by XML namespaces, for example. We already have a central naming authority it's called PyPI ;) > > >> >>> For a while, many of us have been pretty careful to use namespaces >>> for new packages to mitigate this issue. For example, the zc namespace >>> is a shorter version of com.zope, but at some point, it won't be fair >>> for us to claim zc for ourselves. >> >> I wonder if we could allow people/groups to apply (to humans) for a >> namespace which they can subsequently control, like the "zc.*" one. > > We could. Maybe that's the most palatable alternative. It has the > huge benefit of promoting relatively flat, Pythonic, namespaces. > > It has disadvantages: > > - We have to set up some sort of naming authority. > > - It's probably not scalable. As mentioned above we already have an authority, and it's more or less scaled quite fine so far. I still continue to be able to get very good names, but sometimes it takes a few tries to find an unused one. > >> Everyone could continue to push non-namespaced (flat) packages to pypi >> like now but the names couldn't take the form of namespaced ones. > > I'm not sure what you're suggesting here. > > Are you saying someone could publish a package named: "zc", > bit not "zc.foo"? > > Or are you saying that publication of a package named "bar" would > prevent someone from creating a "bar" namespace, and the > other way around? This is likely implementation details, but if I were to do it then folks could apply for a particular namespace, (For example zc.*) and it would give them ownership over all levels of that namespace, (in our example, zc, zc.foo, zc.bar, zc.foo.bar, etc but not zc4 or zc4.cool). However they would only own that namespace if they've applied and been granted it. Otherwise anyone is free to register packages in that namespace. And for what it's worth Organization accounts is something on my roadmap for PyPI. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jaraco at jaraco.com Sun Jun 2 17:42:15 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 2 Jun 2013 15:42:15 +0000 Subject: [Distutils] Setuptools 0.7 final available for download Message-ID: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> I'm pleased to announce the official release of Setuptools 0.7, now available for download from the project page . Nothing has changed from the 0.7b4 pre-release. This is the version that will be uploaded to PyPI after we work out the technique to deploy to PyPI without interfering with the setuptools 0.6 releases. For convenience, I've also added experimental .msi installers for Windows for Python 3.3 and Python 2.7. Work may continue on these in the future, but as the documentation states, the recommended installation procedure is to use ez_setup.py. To install this latest release, follow the canonical install instructions (using ez_setup.py), but direct the script to use bitbucket instead of PyPI: python ./ez_setup.py --download-base=https://bitbucket.org/pypa/setuptools/downloads/ I'll send another announcement when the official release has been uploaded to PyPI and the ez_setup.py script can be used directly. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From donald at stufft.io Sun Jun 2 18:30:00 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 2 Jun 2013 12:30:00 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: <085C44CA-BF20-481E-B8F6-F455FD4BFAB0@stufft.io> On Jun 2, 2013, at 11:42 AM, "Jason R. Coombs" wrote: > I?m pleased to announce the official release of Setuptools 0.7, now available for download from the project page. > > Nothing has changed from the 0.7b4 pre-release. This is the version that will be uploaded to PyPI after we work out the technique to deploy to PyPI without interfering with the setuptools 0.6 releases. > > For convenience, I?ve also added experimental .msi installers for Windows for Python 3.3 and Python 2.7. Work may continue on these in the future, but as the documentation states, the recommended installation procedure is to use ez_setup.py. > > To install this latest release, follow the canonical install instructions (using ez_setup.py), but direct the script to use bitbucket instead of PyPI: > > python ./ez_setup.py --download-base=https://bitbucket.org/pypa/setuptools/downloads/ > > I?ll send another announcement when the official release has been uploaded to PyPI and the ez_setup.py script can be used directly. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Great work! ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From techtonik at gmail.com Sun Jun 2 19:15:01 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 2 Jun 2013 20:15:01 +0300 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: On Sun, Jun 2, 2013 at 6:42 PM, Jason R. Coombs wrote: > I?m pleased to announce the official release of Setuptools 0.7, now > available for download from the project page > .**** > > ** ** > > Nothing has changed from the 0.7b4 pre-release. This is the version that > will be uploaded to PyPI after we work out the technique to deploy to PyPI > without interfering with the setuptools 0.6 releases. > Nice. This is a significant event. Is there any Changes page to see what is so awesome in the release that took about more than 3 years to appear according to PyPI page (https://pypi.python.org/pypi/setuptools)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaraco at jaraco.com Sun Jun 2 19:28:07 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 2 Jun 2013 17:28:07 +0000 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: Hi Paul. From: Paul Moore [mailto:p.f.moore at gmail.com] Sent: Sunday, 02 June, 2013 13:13 On 2 June 2013 16:42, Jason R. Coombs > wrote: For convenience, I've also added experimental .msi installers for Windows for Python 3.3 and Python 2.7. Work may continue on these in the future, but as the documentation states, the recommended installation procedure is to use ez_setup.py. Please consider using wheel instead of msi. Or at least bdist_wininst. The msi format is opaque and cannot be converted to other formats. OTOH, setuptools is pure Python, so having binaries is a relative non-issue. Thanks. I need to learn wheel. I haven't yet done that. The reason I chose msi is because it is a binary distribution format that works on 64-bit Python (there still exists a bug in distutils where installers don't detect 64-bit python installations because the installer executable is 32-bit). The recommended installation technique on all platforms, much like Distribute, is to do a source install. One point - can I assume that the new version is written to run unchanged on all supported Python versions (2 and 3) so that it is possible to build a wheel using *any* version of Python and use it unchanged on any other? (I ask because I'd like to look at integrating setuptools 0.7 into virtualenv). Currently, Python 3 support is still using the same technique employed by distribute - namely, running 2to3 during install. So I don't believe it can be used unchanged on any Python version. Having a single code base for Python 2.4-3.3 is the top feature I want to develop as soon as the merge is complete and successfully adopted. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From p.f.moore at gmail.com Sun Jun 2 19:07:53 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 2 Jun 2013 18:07:53 +0100 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> <20130602125104.Horde.ymzUqA32YbvcCiM0TU_gzg1@webmail.df.eu> Message-ID: On 2 June 2013 12:54, Donald Stufft wrote: > The missing case here is what happens if they don't respond? That (and when there is no author contact details) is when a unilateral process *is* needed. In those cases, I'd argue that we should have some overall guidelines to work from, but ultimately the PyPI admin(s) should decide on a case by case basis. If and when the volume of such requests gets so great that the admins are overwhelmed, *then* we should look at the feasibility of an automatic process. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaraco at jaraco.com Sun Jun 2 19:19:58 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 2 Jun 2013 17:19:58 +0000 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: See https://bitbucket.org/pypa/setuptools/src/0.7/CHANGES.txt and https://bitbucket.org/pypa/setuptools/src/0.7/MERGE.txt for details on the changes. From: anatoly techtonik [mailto:techtonik at gmail.com] Sent: Sunday, 02 June, 2013 13:15 To: Jason R. Coombs Cc: distutils-sig at python.org Subject: Re: [Distutils] Setuptools 0.7 final available for download On Sun, Jun 2, 2013 at 6:42 PM, Jason R. Coombs > wrote: I?m pleased to announce the official release of Setuptools 0.7, now available for download from the project page . Nothing has changed from the 0.7b4 pre-release. This is the version that will be uploaded to PyPI after we work out the technique to deploy to PyPI without interfering with the setuptools 0.6 releases. Nice. This is a significant event. Is there any Changes page to see what is so awesome in the release that took about more than 3 years to appear according to PyPI page (https://pypi.python.org/pypi/setuptools)? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From jim at zope.com Sun Jun 2 19:49:23 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 2 Jun 2013 13:49:23 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: On Sun, Jun 2, 2013 at 1:15 PM, anatoly techtonik wrote: ... > Nice. This is a significant event. Is there any Changes page to see what is > so awesome in the release that took about more than 3 years to appear > according to PyPI page (https://pypi.python.org/pypi/setuptools)? FTR, there have been frequent releases to setuptools over that period. They weren't published to PyPI, but were still seen by setuptools. (I assume through page scraping.) Not ideal, by far, I know. There's a common misconception that setuptools has been dormant, but that's just not accurate. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From tseaver at palladion.com Sun Jun 2 19:51:45 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 02 Jun 2013 13:51:45 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <20130602071018.GH7789@merlinux.eu> References: <20130602071018.GH7789@merlinux.eu> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/02/2013 03:10 AM, holger krekel wrote: > Somewhat proper import namespacing is only available with very recent > python versions which still have a long way to become mainstream. I don't understand this claim at all. W'eve had packages in python for fifteen years, and extensible namespace-package support in one form or another for eight. The fact that the Python 3.3 adds support for a new spelling doesn't mean they are a new feature. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGrhjEACgkQ+gerLs4ltQ7jnQCfYokgj1vHL/pcun2PYmsP6EYD ei4AoJ0AhzbIckRY+mGtIk89qXqaFjY9 =Mqzk -----END PGP SIGNATURE----- From regebro at gmail.com Sun Jun 2 20:49:05 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 2 Jun 2013 20:49:05 +0200 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: Message-ID: On Sat, Jun 1, 2013 at 5:57 PM, Jim Fulton wrote: > In the Python community, we've been pretty laid back > about how we name packages. When we were small, this made > sense. It doesn't make sense any more. I don't think this is a problem, and I don't think domains or usernames in the package names is a solution even if it is a problem. > We should not have to come up with a process for recognizing squatters > on simple package names. We should have something more systematic, > IMO. I also don't think squatting in itself is that much of a problem. Only once has someone been faster than me in stealing a package name and that was "skynet". :-) Pretty much all other package names I've ever come up with has been free. And when somebody is squatting, I think it can be dealt with manually, for the most time. In fact, I'm trying to contact the skynet author now, to see if I can get my "skynet" in there instead. ;-) [1] Something I don't like though is the plethora of non-packages, most of which are test packages of some sort. Just search for "foo". :-) I'd like that to be cleaned up. //Lennart [1] https://github.com/regebro/skynet From techtonik at gmail.com Sun Jun 2 20:50:18 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 2 Jun 2013 21:50:18 +0300 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: On Sun, Jun 2, 2013 at 8:19 PM, Jason R. Coombs wrote: > See https://bitbucket.org/pypa/setuptools/src/0.7/CHANGES.txt and > https://bitbucket.org/pypa/setuptools/src/0.7/MERGE.txt for details on > the changes. > Thanks for the links. I expected something more massive, but as people said there appeared to be a plenty of releases over this period, so MERGE.txt covers my needs, Major number increments mean that some features are added according to semver.org and even without it it is convenient to read about features separately from fixes. http://sourceforge.net/p/roundup/code/ci/default/tree/CHANGES.txt -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaraco at jaraco.com Sun Jun 2 23:14:54 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 2 Jun 2013 21:14:54 +0000 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: Good suggestions. Historically, Setuptools has not followed semver. I intend for Setuptools to start to follow semver (at least in spirit) starting with 0.7. From: anatoly techtonik [mailto:techtonik at gmail.com] Sent: Sunday, 02 June, 2013 14:50 To: Jason R. Coombs Cc: distutils-sig at python.org Subject: Re: [Distutils] Setuptools 0.7 final available for download On Sun, Jun 2, 2013 at 8:19 PM, Jason R. Coombs > wrote: See https://bitbucket.org/pypa/setuptools/src/0.7/CHANGES.txt and https://bitbucket.org/pypa/setuptools/src/0.7/MERGE.txt for details on the changes. Thanks for the links. I expected something more massive, but as people said there appeared to be a plenty of releases over this period, so MERGE.txt covers my needs, Major number increments mean that some features are added according to semver.org and even without it it is convenient to read about features separately from fixes. http://sourceforge.net/p/roundup/code/ci/default/tree/CHANGES.txt -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From jaraco at jaraco.com Sun Jun 2 23:16:42 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 2 Jun 2013 21:16:42 +0000 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: <35606a28d31c4a5e88ef3b0a6457ac05@BY2PR06MB011.namprd06.prod.outlook.com> From: Paul Moore [mailto:p.f.moore at gmail.com] Sent: Sunday, 02 June, 2013 14:03 One other question that I can't see covered by the CHANGES and README files. Does setuptools 0.7 support dist-info directories the same way that recent versions of distribute did? The dist-info support is critical to wheel support in pip. Yes, unless otherwise specified in the merge docs, Setuptools 0.7 includes all of Distribute. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From richard at python.org Sun Jun 2 23:59:37 2013 From: richard at python.org (Richard Jones) Date: Mon, 3 Jun 2013 07:59:37 +1000 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <20130602125104.Horde.ymzUqA32YbvcCiM0TU_gzg1@webmail.df.eu> References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> <20130602125104.Horde.ymzUqA32YbvcCiM0TU_gzg1@webmail.df.eu> Message-ID: I also support human involvement. I don't deal with so many cases that it's such a burden - though I do have a bit of a backlog at the moment due to lack of personal time. Having said that, I can see value in automatically clearing out empty (no meta-data, no files) registrations 6+ months after they're created. On 2 June 2013 20:51, wrote: > > Quoting Paul Moore : > > I'm -1 on anything that doesn't involve at least a minimal level of human >> involvement (possibly excepting an initial clean up exercise for projects >> with no author email) >> > > I support this position. This is actually how PyPI has operated over the > last > decade. People have always taken over projects, either the project > entirely, > or just the name. It always involved contacting the original owner of the > name. > > In this thread, Lukas wrote > > > Fortunately we were able to work it out with Richard >> but we had to contact him directly and waste his cycles on this. >> > > I don't consider his cycles wasted at all. It's an important interaction. > > I'm fine with formalizing the process, and I'm also fine with adding tool > support. However, I agree that a PEP should be written and agreed about > this. > > Personally, I'd favor this procedure: > - nothing happens unless some user explicitly requests it > - on request, the owner is contacted, and given some time to respond > - if they do respond, and are unwilling to yield the name, nothing > happens > - if they have confirmed that they want to keep the name, they won't > be asked again for at least one year. > > Regards, > Martin > > > > > ______________________________**_________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/**mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jun 3 01:52:35 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 3 Jun 2013 09:52:35 +1000 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: Great to hear! Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Mon Jun 3 04:21:02 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 2 Jun 2013 22:21:02 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> Message-ID: On Sat, Jun 1, 2013 at 4:29 PM, Lennart Regebro wrote: > On Sat, Jun 1, 2013 at 9:20 PM, Paul Moore wrote: >> I'm -1 on anything that doesn't involve at least a minimal level of human >> involvement (possibly excepting an initial clean up exercise for projects >> with no author email) > > This is why I basically said I'm OK with automatic deletion after a > time if there are no downloadable packages and no contact information. > Otherwise the owner should be contacted. Some people are saying "files uploaded" vs. "downloadable packages". I don't like the "files uploaded" criterion because IMO it's a perfectly valid use case to list a package on PyPI which is only available via external revision control. Heck, a project that only has planning documents and a reasonably active mailing list should still qualify for PyPI listing, else the original distutils-sig would not have qualified for reserving the name "distutils" on PyPI, before its first release. ;-) From noah at coderanger.net Mon Jun 3 04:43:04 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 2 Jun 2013 19:43:04 -0700 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> Message-ID: <6BE8B4CF-3539-4E84-93AC-644190080D0C@coderanger.net> On Jun 2, 2013, at 7:21 PM, PJ Eby wrote: > On Sat, Jun 1, 2013 at 4:29 PM, Lennart Regebro wrote: >> On Sat, Jun 1, 2013 at 9:20 PM, Paul Moore wrote: >>> I'm -1 on anything that doesn't involve at least a minimal level of human >>> involvement (possibly excepting an initial clean up exercise for projects >>> with no author email) >> >> This is why I basically said I'm OK with automatic deletion after a >> time if there are no downloadable packages and no contact information. >> Otherwise the owner should be contacted. > > Some people are saying "files uploaded" vs. "downloadable packages". > I don't like the "files uploaded" criterion because IMO it's a > perfectly valid use case to list a package on PyPI which is only > available via external revision control. > Sorry, if you haven't had time to follow lately we have already begun deprecating this system. It is entirely reasonable to start making plans for the case when this will no longer be an option. > Heck, a project that only has planning documents and a reasonably > active mailing list should still qualify for PyPI listing, else the > original distutils-sig would not have qualified for reserving the name > "distutils" on PyPI, before its first release. ;-) If a reasonably active project doesn't have anything to show after six months, I think we have different definitions of 'reasonably active'. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From nad at acm.org Mon Jun 3 07:34:29 2013 From: nad at acm.org (Ned Deily) Date: Sun, 02 Jun 2013 22:34:29 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X References: Message-ID: In article , Daniel Holth wrote: > On May 31, 2013 6:30 PM, "Chris Barker - NOAA Federal" < > chris.barker at noaa.gov> wrote: > > On OS-X, binaries can be "universal" -- what this means is that they > > have more than one binary architecture embedded in a single file; the > > system selects the right one at run time. Both executables and dynamic > > libraries can be universal. [...] > Wheel already supports compound tags. Just like py2.py3-none-any a tag > py3-none-x86.ppc (with real platform names) would work. Does that make > sense for Mac? I brought this topic up in passing with Daniel at the PyCon Packaging BOF but haven't gotten around yet to documenting the issues and possible solutions. (Besides the architectures, there is the related issue of deployment target values.) I'll write something up for it this week and post it here. -- Ned Deily, nad at acm.org From regebro at gmail.com Mon Jun 3 07:56:05 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 3 Jun 2013 07:56:05 +0200 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> Message-ID: On Mon, Jun 3, 2013 at 4:21 AM, PJ Eby wrote: > On Sat, Jun 1, 2013 at 4:29 PM, Lennart Regebro wrote: >> On Sat, Jun 1, 2013 at 9:20 PM, Paul Moore wrote: >>> I'm -1 on anything that doesn't involve at least a minimal level of human >>> involvement (possibly excepting an initial clean up exercise for projects >>> with no author email) >> >> This is why I basically said I'm OK with automatic deletion after a >> time if there are no downloadable packages and no contact information. >> Otherwise the owner should be contacted. > > Some people are saying "files uploaded" vs. "downloadable packages". > I don't like the "files uploaded" criterion because IMO it's a > perfectly valid use case to list a package on PyPI which is only > available via external revision control. > > Heck, a project that only has planning documents and a reasonably > active mailing list should still qualify for PyPI listing, else the > original distutils-sig would not have qualified for reserving the name > "distutils" on PyPI, before its first release. ;-) Absolutely. Which gets us back to the "nothing to download, no way of contacting" criteria I originally proposed. :-) //Lennart From chris at simplistix.co.uk Mon Jun 3 08:22:26 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 03 Jun 2013 07:22:26 +0100 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <6BE8B4CF-3539-4E84-93AC-644190080D0C@coderanger.net> References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> <6BE8B4CF-3539-4E84-93AC-644190080D0C@coderanger.net> Message-ID: <51AC3622.9020305@simplistix.co.uk> On 03/06/2013 03:43, Noah Kantrowitz wrote: > >> Some people are saying "files uploaded" vs. "downloadable packages". >> I don't like the "files uploaded" criterion because IMO it's a >> perfectly valid use case to list a package on PyPI which is only >> available via external revision control. > > Sorry, if you haven't had time to follow lately we have already begun deprecating this system. I hope you're misunderstanding what pje is saying; this isn't about hosting distributions elsewhere, this is about having a PyPI listing for a project that is under development but it hasn't got to the point where it's sensible for a release to be made. >> Heck, a project that only has planning documents and a reasonably >> active mailing list should still qualify for PyPI listing, else the >> original distutils-sig would not have qualified for reserving the name >> "distutils" on PyPI, before its first release. ;-) > > If a reasonably active project doesn't have anything to show after six months, I think we have different definitions of 'reasonably active'. ...or different definitions of 'software quality' ;-) Seriously, I don't think anyone would argue that we have enough nested list printers in this world or that a package without any contact details or description is fair game to delete, but I must echo what others have said in that a unilateral process where a project is deleted without the owner being given a reasonable time to respond doesn't seem like a good idea. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From donald at stufft.io Mon Jun 3 09:13:45 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 03:13:45 -0400 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: <51AC3622.9020305@simplistix.co.uk> References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> <6BE8B4CF-3539-4E84-93AC-644190080D0C@coderanger.net> <51AC3622.9020305@simplistix.co.uk> Message-ID: On Jun 3, 2013, at 2:22 AM, Chris Withers wrote: > On 03/06/2013 03:43, Noah Kantrowitz wrote: >> >>> Some people are saying "files uploaded" vs. "downloadable packages". >>> I don't like the "files uploaded" criterion because IMO it's a >>> perfectly valid use case to list a package on PyPI which is only >>> available via external revision control. >> >> Sorry, if you haven't had time to follow lately we have already begun deprecating this system. > > I hope you're misunderstanding what pje is saying; this isn't about hosting distributions elsewhere, this is about having a PyPI listing for a project that is under development but it hasn't got to the point where it's sensible for a release to be made. > >>> Heck, a project that only has planning documents and a reasonably >>> active mailing list should still qualify for PyPI listing, else the >>> original distutils-sig would not have qualified for reserving the name >>> "distutils" on PyPI, before its first release. ;-) >> >> If a reasonably active project doesn't have anything to show after six months, I think we have different definitions of 'reasonably active'. > > ...or different definitions of 'software quality' ;-) > > Seriously, I don't think anyone would argue that we have enough nested list printers in this world or that a package without any contact details or description is fair game to delete, but I must echo what others have said in that a unilateral process where a project is deleted without the owner being given a reasonable time to respond doesn't seem like a good idea. Unless I missed it I don't think I've seen *anyone* suggest that projects where the authors appear to actually have plans to use the name have the name yanked out from underneath them. > > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris at simplistix.co.uk Mon Jun 3 09:35:27 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 03 Jun 2013 08:35:27 +0100 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> Message-ID: <51AC473F.609@simplistix.co.uk> On 03/06/2013 06:56, Lennart Regebro wrote: > On Mon, Jun 3, 2013 at 4:21 AM, PJ Eby wrote: >> On Sat, Jun 1, 2013 at 4:29 PM, Lennart Regebro wrote: >>> On Sat, Jun 1, 2013 at 9:20 PM, Paul Moore wrote: >>>> I'm -1 on anything that doesn't involve at least a minimal level of human >>>> involvement (possibly excepting an initial clean up exercise for projects >>>> with no author email) >>> >>> This is why I basically said I'm OK with automatic deletion after a >>> time if there are no downloadable packages and no contact information. >>> Otherwise the owner should be contacted. >> >> Some people are saying "files uploaded" vs. "downloadable packages". >> I don't like the "files uploaded" criterion because IMO it's a >> perfectly valid use case to list a package on PyPI which is only >> available via external revision control. >> >> Heck, a project that only has planning documents and a reasonably >> active mailing list should still qualify for PyPI listing, else the >> original distutils-sig would not have qualified for reserving the name >> "distutils" on PyPI, before its first release. ;-) > > Absolutely. Which gets us back to the "nothing to download, no way of > contacting" criteria I originally proposed. :-) +1 :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From holger at merlinux.eu Mon Jun 3 09:35:31 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 3 Jun 2013 07:35:31 +0000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130602071018.GH7789@merlinux.eu> Message-ID: <20130603073531.GN7789@merlinux.eu> On Sun, Jun 02, 2013 at 13:51 -0400, Tres Seaver wrote: > On 06/02/2013 03:10 AM, holger krekel wrote: > > Somewhat proper import namespacing is only available with very recent > > python versions which still have a long way to become mainstream. > > I don't understand this claim at all. W'eve had packages in python for > fifteen years, and extensible namespace-package support in one form or > another for eight. The fact that the Python 3.3 adds support for a new > spelling doesn't mean they are a new feature. I stand corrected. To be honest, i didn't consider the setuptools extensible namespace support a proper solution but indeed it exists and is used. best, holger > > Tres. > -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From chris at simplistix.co.uk Mon Jun 3 09:36:39 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 03 Jun 2013 08:36:39 +0100 Subject: [Distutils] A process for removal of PyPi entries In-Reply-To: References: <20130531130939.GC7789@merlinux.eu> <51AA0C75.6070805@simplistix.co.uk> <6BE8B4CF-3539-4E84-93AC-644190080D0C@coderanger.net> <51AC3622.9020305@simplistix.co.uk> Message-ID: <51AC4787.9070901@simplistix.co.uk> On 03/06/2013 08:13, Donald Stufft wrote: > >>> If a reasonably active project doesn't have anything to show after >>> six months, I think we have different definitions of 'reasonably active'. >> >> ...or different definitions of 'software quality' ;-) >> >> Seriously, I don't think anyone would argue that we have enough nested >> list printers in this world or that a package without any contact >> details or description is fair game to delete, but I must echo what >> others have said in that a unilateral process where a project is >> deleted without the owner being given a reasonable time to respond >> doesn't seem like a good idea. > > Unless I missed it I don't think I've seen *anyone* suggest that > projects where the authors appear to actually have plans to use the name > have the name yanked out from underneath them. ...in that case, I'm happy to find out I have misunderstood :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From holger at merlinux.eu Mon Jun 3 09:57:32 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 3 Jun 2013 07:57:32 +0000 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: References: <20130601192112.GG7789@merlinux.eu> Message-ID: <20130603075732.GO7789@merlinux.eu> Hi Jim, On Sun, Jun 02, 2013 at 11:25 -0400, Jim Fulton wrote: > On Sat, Jun 1, 2013 at 3:21 PM, holger krekel wrote: > > On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: > >> In the Python community, we've been pretty laid back > >> about how we name packages. When we were small, this made > >> sense. It doesn't make sense any more. > > > > I've heart this sentiment before, but would like to read more > > clearly stated problems. > > I thought the problem was pretty clear: name collisions. In theory yes, but is a practical problem? Similar to Donald, so far i didn't find myself or hear much from other people that they have problems finding good pypi names. We have some 30K names registered with PyPI i think -- and many top level DNS entries have millions of names. So the numbers don't clearly imply we have a scaling problem. > There's a parallel thread on how to detect and reclaim > names that are taken and unused. I think if we had a more > systematic way of naming packages, this wouldn't be an issue. It might still be an issue. > ... > > Everyone could continue to push non-namespaced (flat) packages to pypi > > like now but the names couldn't take the form of namespaced ones. > > I'm not sure what you're suggesting here. > > Are you saying someone could publish a package named: "zc", > bit not "zc.foo"? > > Or are you saying that publication of a package named "bar" would > prevent someone from creating a "bar" namespace, and the > other way around? I'd think that once a group registers a "X" namespace and it is accepted it would give control to registering any "X.*" packages to that group. As a pre-condition there shouldn't be any bare "X" package existing i guess. Clearly, the exact semantics/process would need to be thought out. I'd like to base further discussion on actual PEP drafts with a proper statement of practical problems. What i personally would consider a practical problem is this: "As a company/community we have already N packages and we want to communicate our users through pypi names that using the namespace 'XY.' will always get them software that we have screened/reviewed/released ourselves. But currently there is no way to prevent others from releasing "ZY." prefixed software". cheers, holger > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > From d.s.seljebotn at astro.uio.no Mon Jun 3 09:48:06 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 03 Jun 2013 09:48:06 +0200 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <20130603073531.GN7789@merlinux.eu> References: <20130602071018.GH7789@merlinux.eu> <20130603073531.GN7789@merlinux.eu> Message-ID: <51AC4A36.8010904@astro.uio.no> On 06/03/2013 09:35 AM, holger krekel wrote: > On Sun, Jun 02, 2013 at 13:51 -0400, Tres Seaver wrote: >> On 06/02/2013 03:10 AM, holger krekel wrote: >>> Somewhat proper import namespacing is only available with very recent >>> python versions which still have a long way to become mainstream. >> >> I don't understand this claim at all. W'eve had packages in python for >> fifteen years, and extensible namespace-package support in one form or >> another for eight. The fact that the Python 3.3 adds support for a new >> spelling doesn't mean they are a new feature. > > I stand corrected. To be honest, i didn't consider the setuptools > extensible namespace support a proper solution but indeed it exists > and is used. FWIW, the small scientific libraries associated with the SciPy ecosystem was gathered under the "scikits" umbrella, so that you had "scikits.learn", "scikits.image", and so on. Eventually the setuptools pain made them decide to drop that and name themselves "sklearn", "skimage". Dag Sverre From jim at zope.com Mon Jun 3 12:08:01 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 3 Jun 2013 06:08:01 -0400 Subject: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. In-Reply-To: <20130603073531.GN7789@merlinux.eu> References: <20130602071018.GH7789@merlinux.eu> <20130603073531.GN7789@merlinux.eu> Message-ID: On Mon, Jun 3, 2013 at 3:35 AM, holger krekel wrote: > On Sun, Jun 02, 2013 at 13:51 -0400, Tres Seaver wrote: >> On 06/02/2013 03:10 AM, holger krekel wrote: >> > Somewhat proper import namespacing is only available with very recent >> > python versions which still have a long way to become mainstream. >> >> I don't understand this claim at all. W'eve had packages in python for >> fifteen years, and extensible namespace-package support in one form or >> another for eight. The fact that the Python 3.3 adds support for a new >> spelling doesn't mean they are a new feature. > > I stand corrected. To be honest, i didn't consider the setuptools > extensible namespace support a proper solution but indeed it exists > and is used. Before the setuptools namespace support, there was the pkgutil module in the standard library. Setuptools' system only added support for zipped packages. Python's current package system allows you to manipulate a package's path, used to find modules and sub-packages in the package. The Zope project used this mechanism for years support the Zope "Products" namespace package. The pkgutils module, and later setuptools' mechanism just simplified this. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From tseaver at palladion.com Mon Jun 3 14:44:47 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 03 Jun 2013 08:44:47 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/02/2013 11:42 AM, Jason R. Coombs wrote: > I'm pleased to announce the official release of Setuptools 0.7, now > available for download from the project page > . > > > > Nothing has changed from the 0.7b4 pre-release. This is the version > that will be uploaded to PyPI after we work out the technique to > deploy to PyPI without interfering with the setuptools 0.6 releases. > > > > For convenience, I've also added experimental .msi installers for > Windows for Python 3.3 and Python 2.7. Work may continue on these in > the future, but as the documentation states, the recommended > installation procedure is to use ez_setup.py. > > > > To install this latest release, follow the canonical install > instructions (using ez_setup.py), but direct the script to use > bitbucket instead of PyPI: Maybe I'm missing something, but where is the 'ez_zetup.py' script hosted? It isn't present in the bitbucket downloads directlry. https://bitbucket.org/pypa/setuptools/downloads/ Maybe linking to or recapitulating the "canonical install instructions" here would help diespel confusion. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGsj78ACgkQ+gerLs4ltQ4JqQCgrSWCnecjtDTk5BrupmQEEUkB l/YAn3qY9uxe1Fb4PXPpOnIV0wgWy4Rd =/7uv -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Jun 3 14:50:27 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 03 Jun 2013 08:50:27 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/03/2013 08:44 AM, Tres Seaver wrote: > On 06/02/2013 11:42 AM, Jason R. Coombs wrote: >> I'm pleased to announce the official release of Setuptools 0.7, now >> available for download from the project page >> . > > > >> Nothing has changed from the 0.7b4 pre-release. This is the version >> that will be uploaded to PyPI after we work out the technique to >> deploy to PyPI without interfering with the setuptools 0.6 >> releases. > > > >> For convenience, I've also added experimental .msi installers for >> Windows for Python 3.3 and Python 2.7. Work may continue on these >> in the future, but as the documentation states, the recommended >> installation procedure is to use ez_setup.py. > > > >> To install this latest release, follow the canonical install >> instructions (using ez_setup.py), but direct the script to use >> bitbucket instead of PyPI: > > Maybe I'm missing something, but where is the 'ez_zetup.py' script > hosted? It isn't present in the bitbucket downloads directlry. > > https://bitbucket.org/pypa/setuptools/downloads/ > > Maybe linking to or recapitulating the "canonical install > instructions" here would help diespel confusion. I tried looking at the README on Bitbucket: https://bitbucket.org/pypa/setuptools/#rst-header-installation-instructions but that points to an invalid URL for 'ez_setup.py': https://bitbucket.org/pypa/setuptools/raw/0.7.1/ez_setup.py The version here: https://bitbucket.org/pypa/setuptools/raw/0.7/ez_setup.py points to PyPI for the DEFAULT_URL. During the beta, I think we need to have a version ov 'ez_setup.py' available which points to the Bitbucket download page, rather than PyPI. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGskRMACgkQ+gerLs4ltQ45IQCdEFrOuCszGKStzKxKPdN2vPd/ /fMAn2j5mdBhE4PthXlG/wVVqv1ztoz9 =kFhy -----END PGP SIGNATURE----- From christian at python.org Mon Jun 3 15:04:13 2013 From: christian at python.org (Christian Heimes) Date: Mon, 03 Jun 2013 15:04:13 +0200 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: <51AC944D.8000004@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Am 03.06.2013 14:44, schrieb Tres Seaver: > Maybe I'm missing something, but where is the 'ez_zetup.py' script > hosted? It isn't present in the bitbucket downloads directlry. > > https://bitbucket.org/pypa/setuptools/downloads/ > > Maybe linking to or recapitulating the "canonical install > instructions" here would help diespel confusion. It's here https://bitbucket.org/pypa/setuptools/src . Another concern: ez_zetup.py neither uses https to download setuptools nor contains checksums anymore. setuptools 0.6 at least used to validate the download with a known MD5 checksum. I suggest that SHA-1 hashsums are added to ez_setup.py. SHA-2 is not supported by Python < 2.5. Christian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCAAGBQJRrJRAAAoJEMeIxMHUVQ1FVicQAIc7F52TWwIK6WOPGpvimozN fe9JnJI0nCtgCgaplOnneS9JR82GJOOX1vOWJyapo/q2xGiT+79gHV38vkxv50f8 C22B2sfA5K4kgQutqNnZVYzwVFb5f6idR8lTE8D2Zkwt6CUsEyeJlAMfG8rkL903 UcjSDgsTcPFsGXPDvkw9vfirrJ1aQQcVpbK4NlN4Xn5uGN5vpvxmVvBrAYRZ/1nX L9l81I/1njCJmNgXRxOGdPZAxZtkrzgW1tNc/hZxKNeXhN4mO7RQwbYwjDtCcy8F gmxeo4jK/nFrGP71bQTH55erFAIva8rWPl7J3M5W2l9fnrn4Iv80oScv5wSOAcW6 EP53pS1RPT1IEtHI3TyADicqz8pyuj3aLSiih0H62YhBLIDWJ/WZ4PAWoL2hbSXN aveoqqpjXVrjH+UNhs1Zy27ns8AY2sgwKiyeOwji1U/uICAl9hsjmDZLg6110z71 q3arNPQOdnRdLUYO8uX6n4BFdjlqtv070Z6plXeOHQSpMC4Yzypz4b3l+6dvc00p hFcJpmecvaS052QGRANRy+rqKHJ7AgGd39LjKgFetQ56mNWLAPfqNafPqxRit8y7 jyCjehaNjFHX5VImxK3mU+dl+q2+eCNbZ1BA+YZ/aWlqlZ8FD+6LwNMuAVy5RqZ3 SGV/gwwGdufzcMbzEEJv =GLKv -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Jun 3 15:14:04 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 03 Jun 2013 09:14:04 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: <51AC944D.8000004@python.org> References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> <51AC944D.8000004@python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/03/2013 09:04 AM, Christian Heimes wrote: > It's here https://bitbucket.org/pypa/setuptools/src . That version points to PyPI as the location, and wants to install version 0.7.1. Until 0.7 is pushed to PyPI, we need an ez_setup.py available which points to the bitbucket download page. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGslpwACgkQ+gerLs4ltQ7IRwCfS1suMO0YzA9P1iTvNDDd+Hrm myoAn0uqw7cyOuqqaHj/DaXO5sRA53BY =rFuY -----END PGP SIGNATURE----- From erik.m.bray at gmail.com Mon Jun 3 16:16:49 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 3 Jun 2013 10:16:49 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: On Sun, Jun 2, 2013 at 11:42 AM, Jason R. Coombs wrote: > I?m pleased to announce the official release of Setuptools 0.7, now > available for download from the project page. > > > > Nothing has changed from the 0.7b4 pre-release. This is the version that > will be uploaded to PyPI after we work out the technique to deploy to PyPI > without interfering with the setuptools 0.6 releases. > > > > For convenience, I?ve also added experimental .msi installers for Windows > for Python 3.3 and Python 2.7. Work may continue on these in the future, but > as the documentation states, the recommended installation procedure is to > use ez_setup.py. > > > > To install this latest release, follow the canonical install instructions > (using ez_setup.py), but direct the script to use bitbucket instead of PyPI: > > > > python ./ez_setup.py > --download-base=https://bitbucket.org/pypa/setuptools/downloads/ > > > > I?ll send another announcement when the official release has been uploaded > to PyPI and the ez_setup.py script can be used directly. Thank you Jason and PJ for all your work on this. It's maybe a little late to bring this up now, but I wonder if, in a future release, we could deprecate 'ez_setup.py' (venerable though it is) and name it something a little more obvious, like 'setuptools_setup.py'. Despite being longer and including the word 'setup' twice I think it is a great deal more clear what it does. I remember in the past, before I switched to distribute, I got reports from users who were running the ez_setup.py that I included in projects thinking that it was the "easy" way to install my software. Yes, they should have just read the README that says "python setup.py install" and makes no mention of ez_setup.py, but nevertheless I can't blame anyone for finding it vague (and maybe even a bit enticing to run). Erik From tseaver at palladion.com Mon Jun 3 16:29:04 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 03 Jun 2013 10:29:04 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> <51AC944D.8000004@python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/03/2013 09:14 AM, Tres Seaver wrote: > Until 0.7 is pushed to PyPI, we need an ez_setup.py available which > points to the bitbucket download page. As an alternative, we could just go ahead and push out this version to the cheeseshop: AFAICT, it doesn't break stuff (especially compared to the current 0.6 nightly -- see issue #8[1]), and anybody who needs to can pin 'setuptools<0.7dev'. [1] https://bitbucket.org/pypa/setuptools/issue/8/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGsqDAACgkQ+gerLs4ltQ5LHQCdHrCQf4rtvumUvROUJLihdEXC YvEAn3MIY3o2YYxm+X42santI6fTtXC+ =hsyB -----END PGP SIGNATURE----- From setuptools at bugs.python.org Mon Jun 3 17:25:48 2013 From: setuptools at bugs.python.org (Tres Seaver) Date: Mon, 03 Jun 2013 15:25:48 +0000 Subject: [Distutils] [issue150] ssl_support.py: 1868:04732ee16e7e broke '_dnsname_to_pat' Message-ID: <1370273148.5.0.192770655981.issue150@psf.upfronthosting.co.za> New submission from Tres Seaver: That Hg commit reference dropped the import of 're', which breaks the '_dnsname_to_pat' fallback function. I reported this on bitbucket: https://bitbucket.org/pypa/setuptools/issue/8 and pushed changes for a pull request there: https://bitbucket.org/pypa/setuptools/pull-request/4 but Jason thinks the 0.6 branch is still primarily managed here and in SVN. The bug breaks the zope.org buildouts, which are still using the 0.6 nightlies while the 0.7 release waits in the wings. ---------- assignedto: pje messages: 721 nosy: pje, tseaver priority: urgent status: unread title: ssl_support.py: 1868:04732ee16e7e broke '_dnsname_to_pat' _______________________________________________ Setuptools tracker _______________________________________________ From chris.barker at noaa.gov Mon Jun 3 18:25:41 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 3 Jun 2013 09:25:41 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: On Fri, May 31, 2013 at 3:38 PM, Daniel Holth wrote: > Wheel already supports compound tags. Just like py2.py3-none-any a tag Is this from PEP 425? It's still a little unclear to me exactly how those tags are to be used, but, yes, this seems to be the right direction. > py3-none-x86.ppc (with real platform names) would work. Does that make sense > for Mac? hmm, for one, I"m confused about the "Python Tag" vs. the "ABI Tag" vs. the "Platform Tag" -- I guess these all get mushed together in the filename? but a few issues anyway: it looks like the platform tag has the OS and a architecture in one, so would a universal bild on the mac be: macosx_10_3_i386.ppc or macosx_10_3_i386.macosx_10_3_ppc I'm thinking the latter, though it gets pretty wordy: A potential full name: my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl (Im a little confused about python version tag vs. ABI tag, in the case of a compiled extension...) But a potential confusion -- in this case, having two platform tags indicates that _both_ are present, rather than one thing that works with either. Maybe that would always be the case with platform. So if that's how the wheel is specified, we still need the other side of the process: Installing: A "universal" macpython binary needs to know that it is universal (distutils.util.get_platform() returns only the currently running version). Once that is know, the installer needs to make some choice: If a matching universal binary is found -- use it If a non-universal binary is found that matches the currently running python -- then what? install it with a warning? raise an exception? -Chris > > On May 31, 2013 6:30 PM, "Chris Barker - NOAA Federal" > wrote: >> >> HI Folks, >> >> A few of us over on the pythonmac list have been hashing out how best >> to support binary packages on OS-X. The binary wheel option seems very >> promising. >> >> However, there is one Mac-specific issue that does not seem to be >> addressed: >> >> On OS-X, binaries can be "universal" -- what this means is that they >> have more than one binary architecture embedded in a single file; the >> system selects the right one at run time. Both executables and dynamic >> libraries can be universal. In theory, an universal binary can >> currently contain up to 4 architectures: >> >> 32+64 bit PPC and 32+64bit x86 >> >> In practice, 64 bit PPC was never broadly used, and 32bit PPC is >> fading fast. Nevertheless, both 32 and 64 Intel systems are pretty >> common, and who knows what there may be in the future (you never know >> with Apple...) >> >> The binary builds of Python served up on the python.org web site have >> supported universal builds for years -- currently there are two >> options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it >> turns out it's hard to support both up to date x86_64 and PPC with the >> same compiler. In these builds, the configuration is set up so that >> distutils will build universal extensions that match the architectures >> included in the builds. This makes it pretty easy to build binary >> packages, and/or full app distributions (py2app) that are universal as >> well. >> >> setuptools' easy_install supports binary eggs, but always choked on >> universal builds -- it didn't know how to identify what matched the >> system. It would be really nice if future tools could identify and >> install the correct binary wheels for universal builds. >> >> I've taken a look at PEP 427, and see this: >> >> platform tag >> E.g. 'linux_x86_64', 'any'. >> >> That looks to me like there is a built-in assumption that a wheel is >> either specific to a single architecture, or any -- so no way to >> specify that it may have multiple architectures. So I propose that >> platform tag allow a list of some sort: >> >> ['macosx-10_6_i386', 'macosx_10_6-x86_64'] >> >> or, I suppose to keep it simple, a single string: >> >> 'macosx_10_6_i386+macosx_10_6_x86_64' >> or >> 'macosx_10_6_i386+x86_64' >> >> If so, then a binary wheel could be built (and discovered) that >> supported multiple architectures. >> >> Then how would an installer ( pip? ) identify the right one? This is a >> bit tricky. PEP 425 states: >> >> "The platform tag is simply distutils.util.get_platform() with all >> hyphens - and periods . replaced with underscore" >> >> On my system right now, I have 32bitPPC + 32bit i386 universal python, >> running on an intel system: >> >> In [2]: distutils.util.get_platform() >> Out[2]: 'macosx-10.3-i386' >> >> So we may want to patch get_platform() to support universal builds - >> or add anther function or something -- you may sometimes what to know >> what you are running right now, and other times want to know how the >> current python is built. (note that while distutils can build >> universal binaries, it's not very smart about it -- all is does is >> grab the compiler and linker flags from the Makefile, which include >> multiple "-arch" flags. >> >> Even if there is an easy way to query the system, that leaves the >> question of what to do. If a user wants to intall a package into a >> universal python, will the installer only accept a binary wheel with >> all the supported architectures? or one with only the currently >> running one? >> >> Anyway, it would be nice to be able to get this stuff to "just work" >> with universal binaries on the Mac. >> >> Thoughts, ideas? >> >> -Chris >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From chris.barker at noaa.gov Mon Jun 3 18:36:24 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 3 Jun 2013 09:36:24 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: And Ned, Thanks for your offer to write something up -- looking forward to it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dholth at gmail.com Mon Jun 3 18:57:05 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 3 Jun 2013 12:57:05 -0400 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: On Mon, Jun 3, 2013 at 12:25 PM, Chris Barker - NOAA Federal wrote: > On Fri, May 31, 2013 at 3:38 PM, Daniel Holth wrote: >> Wheel already supports compound tags. Just like py2.py3-none-any a tag > > Is this from PEP 425? It's still a little unclear to me exactly how > those tags are to be used, but, yes, this seems to be the right > direction. > >> py3-none-x86.ppc (with real platform names) would work. Does that make sense >> for Mac? > > hmm, for one, I"m confused about the "Python Tag" vs. the "ABI Tag" > vs. the "Platform Tag" -- I guess these all get mushed together in the > filename? > > but a few issues anyway: it looks like the platform tag has the OS and > a architecture in one, so would a universal bild on the mac be: > > macosx_10_3_i386.ppc > or > macosx_10_3_i386.macosx_10_3_ppc > > I'm thinking the latter, though it gets pretty wordy: > > A potential full name: > > my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl > > (Im a little confused about python version tag vs. ABI tag, in the > case of a compiled extension...) > > But a potential confusion -- in this case, having two platform tags > indicates that _both_ are present, rather than one thing that works > with either. Maybe that would always be the case with platform. > > So if that's how the wheel is specified, we still need the other side > of the process: > > Installing: > > A "universal" macpython binary needs to know that it is universal > (distutils.util.get_platform() returns only the currently running > version). Once that is know, the installer needs to make some choice: > > If a matching universal binary is found -- use it > If a non-universal binary is found that matches the currently running > python -- then what? install it with a warning? raise an exception? > > -Chris The Python Tag is the Python implementation or language version. It lets you indicate compatibility with only CPython or Jython or PyPy even if you do not have compiled extensions: cp27, py27, py2, py3 The ABI tag is the Python ABI. The difference is that it will indicate "d"ebug "m"alloc, "u"nicode: none (also means "we don't know"), cp33dm, abi3 The platform tag is the OS+architecture. They get joined together with a dash '-'.join(python, abi, platform) For compatibility matching the tags are expanded to the Cartesian product of the string.split('.') of each of the python, abi, and platform tags. This is mostly only useful when only one part of the tag is compound. http://www.python.org/dev/peps/pep-0425/#compressed-tag-sets There used to be a rule about preferring packages with the fewest number of tags in case of a tie but I can't seem to find it now. It's probably not very important. If this doesn't work then it might be more useful to invent a tag like "macosx_10_3_universal" for some value of universal. Daniel Holth From chris.barker at noaa.gov Tue Jun 4 00:56:50 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 3 Jun 2013 15:56:50 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: Darn! this slipped off the list -- I hate it when lists aren't set with the reply-to address as the list... On Mon, Jun 3, 2013 at 3:47 PM, Daniel Holth wrote: > ppc is not completely dead? I have one under my desk, though truth be told, I haven't turned it on in a good while... Which is a good point, of course, we really only have one type of "universal" build to support, and even that one may be obsolete before too long -- not sure how many 32bit only macs are still around. > That would make sense. Can you come up with code to detect that a > newly compiled extension is universal, and that a Python is? can it be platform-dependent? i.e. use a mac-only system call? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dholth at gmail.com Tue Jun 4 01:03:04 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 3 Jun 2013 19:03:04 -0400 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: On Mon, Jun 3, 2013 at 6:56 PM, Chris Barker - NOAA Federal wrote: > Darn! this slipped off the list -- I hate it when lists aren't set > with the reply-to address as the list... > > On Mon, Jun 3, 2013 at 3:47 PM, Daniel Holth wrote: >> ppc is not completely dead? > > I have one under my desk, though truth be told, I haven't turned it on > in a good while... > > Which is a good point, of course, we really only have one type of > "universal" build to support, and even that one may be obsolete > before too long -- not sure how many 32bit only macs are still around. > >> That would make sense. Can you come up with code to detect that a >> newly compiled extension is universal, and that a Python is? > > can it be platform-dependent? i.e. use a mac-only system call? Of course. From chris.barker at noaa.gov Tue Jun 4 00:57:37 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 3 Jun 2013 15:57:37 -0700 Subject: [Distutils] Fwd: Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: Folks, Accidentally let this slip off the list... -Chris ---------- Forwarded message ---------- From: Chris Barker - NOAA Federal Date: Mon, Jun 3, 2013 at 1:33 PM Subject: Re: [Distutils] Binary Wheels and "universal" builds on OS-X To: Daniel Holth On Mon, Jun 3, 2013 at 9:57 AM, Daniel Holth wrote: >> A potential full name: >> >> my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl >> >> (Im a little confused about python version tag vs. ABI tag, in the >> case of a compiled extension...) > The Python Tag is the Python implementation or language version. It > lets you indicate compatibility with only CPython or Jython or PyPy > even if you do not have compiled extensions: so, in this case, is is right to put cp27 in there twice? > The platform tag is the OS+architecture. yeah, kind of wish these were separate, but not really a big deal. > For compatibility matching the tags are expanded to the Cartesian > product of the string.split('.') of each of the python, abi, and > platform tags. This is mostly only useful when only one part of the > tag is compound. > http://www.python.org/dev/peps/pep-0425/#compressed-tag-sets OK -- I think I've got it... > If this doesn't work then it might be more useful to invent a tag like > "macosx_10_3_universal" for some value of universal. hmm -- the trick is that "universal" implies that the binary is, well, universal, but what t really means is "has more than one architecture in it". In practice, there are only a few combinations likely to see use, so maybe just defining a few (only two, now, on the python.org site...) tags makes sense: macosx_10_3_i386_ppc and macosx_10_6_i386_x86_64 -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From regebro at gmail.com Tue Jun 4 04:43:16 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 4 Jun 2013 04:43:16 +0200 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> <51AC944D.8000004@python.org> Message-ID: On Mon, Jun 3, 2013 at 4:29 PM, Tres Seaver wrote: > As an alternative, we could just go ahead and push out this version to > the cheeseshop: AFAICT, it doesn't break stuff (especially compared to > the current 0.6 nightly -- see issue #8[1]), and anybody who needs to can > pin 'setuptools<0.7dev'. +1 From donald at stufft.io Tue Jun 4 07:46:52 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 01:46:52 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail Message-ID: Just want to apologize to anyone who recently received a lot of mail from me. I realized shortly after I queued all the emails that they should have been grouped by user and not by package. So, I'm sorry. I really am and I feel bad about the amount of mail some of you have received from me. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris at python.org Tue Jun 4 08:29:17 2013 From: chris at python.org (Chris Withers) Date: Tue, 04 Jun 2013 07:29:17 +0100 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: <51AD893D.1020502@python.org> Please can you do something to stop it? Kill the MTA or something? This is ridiculous... Chris On 04/06/2013 06:46, Donald Stufft wrote: > Just want to apologize to anyone who recently received a lot of mail > from me. > > I realized shortly after I queued all the emails that they should have > been grouped by user and not by package. > > So, I'm sorry. I really am and I feel bad about the amount of mail some > of you have received from me. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________ > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From techtonik at gmail.com Tue Jun 4 08:29:43 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 4 Jun 2013 09:29:43 +0300 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: That's ok. The bigger problem is to understand English text. =) -- anatoly t. On Tue, Jun 4, 2013 at 8:46 AM, Donald Stufft wrote: > Just want to apologize to anyone who recently received a lot of mail from > me. > > I realized shortly after I queued all the emails that they should have > been grouped by user and not by package. > > So, I'm sorry. I really am and I feel bad about the amount of mail some of > you have received from me. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Tue Jun 4 08:33:22 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 3 Jun 2013 23:33:22 -0700 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <51AD893D.1020502@python.org> References: <51AD893D.1020502@python.org> Message-ID: On Jun 3, 2013, at 11:29 PM, Chris Withers wrote: > Please can you do something to stop it? > Kill the MTA or something? > > This is ridiculous? As someone also in the top percentile of package maintainers I understand your annoyance, but just make a filter for donald at python.org for the day or something. The vast majority of PyPI users have only one package so asking us to derail the sending (probably resulting in having to begin again) is unhelpful. If the only cost to us all is hitting Ctrl-A Delete, I welcome progress with open arms. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris at python.org Tue Jun 4 08:37:47 2013 From: chris at python.org (Chris Withers) Date: Tue, 04 Jun 2013 07:37:47 +0100 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: <51AD893D.1020502@python.org> Message-ID: <51AD8B3B.4070708@python.org> On 04/06/2013 07:33, Noah Kantrowitz wrote: > > On Jun 3, 2013, at 11:29 PM, Chris Withers wrote: > >> Please can you do something to stop it? >> Kill the MTA or something? >> >> This is ridiculous? > > As someone also in the top percentile of package maintainers I understand your annoyance, but just make a filter for donald at python.org for the day or something. The vast majority of PyPI users have only one package so asking us to derail the sending (probably resulting in having to begin again) is unhelpful. If the only cost to us all is hitting Ctrl-A Delete, I welcome progress with open arms. Are you not concerned that various bits of python.org involved in this process are going to start getting hit by RBLs and other spam filtering that will cause problems down the line as a result of all this noise? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From donald at stufft.io Tue Jun 4 08:41:04 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 02:41:04 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <51AD8B3B.4070708@python.org> References: <51AD893D.1020502@python.org> <51AD8B3B.4070708@python.org> Message-ID: <6CED137F-4134-4227-AF84-8330731B7D05@stufft.io> On Jun 4, 2013, at 2:37 AM, Chris Withers wrote: > On 04/06/2013 07:33, Noah Kantrowitz wrote: >> >> On Jun 3, 2013, at 11:29 PM, Chris Withers wrote: >> >>> Please can you do something to stop it? >>> Kill the MTA or something? >>> >>> This is ridiculous? >> >> As someone also in the top percentile of package maintainers I understand your annoyance, but just make a filter for donald at python.org for the day or something. The vast majority of PyPI users have only one package so asking us to derail the sending (probably resulting in having to begin again) is unhelpful. If the only cost to us all is hitting Ctrl-A Delete, I welcome progress with open arms. > > Are you not concerned that various bits of python.org involved in this process are going to start getting hit by RBLs and other spam filtering that will cause problems down the line as a result of all this noise? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Honestly at this point if anything like that happens the damage is likely already done. The vast bulk of them have already been sent. Starting over now would be more emails than letting them finish. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Tue Jun 4 08:45:03 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 3 Jun 2013 23:45:03 -0700 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <51AD8B3B.4070708@python.org> References: <51AD893D.1020502@python.org> <51AD8B3B.4070708@python.org> Message-ID: On Jun 3, 2013, at 11:37 PM, Chris Withers wrote: > On 04/06/2013 07:33, Noah Kantrowitz wrote: >> >> On Jun 3, 2013, at 11:29 PM, Chris Withers wrote: >> >>> Please can you do something to stop it? >>> Kill the MTA or something? >>> >>> This is ridiculous? >> >> As someone also in the top percentile of package maintainers I understand your annoyance, but just make a filter for donald at python.org for the day or something. The vast majority of PyPI users have only one package so asking us to derail the sending (probably resulting in having to begin again) is unhelpful. If the only cost to us all is hitting Ctrl-A Delete, I welcome progress with open arms. > > Are you not concerned that various bits of python.org involved in this process are going to start getting hit by RBLs and other spam filtering that will cause problems down the line as a result of all this noise? No, because only a very small number of people are going to be getting more than a handful of these, and any user with 50+ packages is hopefully enough of a power-user to not fly off the handle. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Jun 4 08:49:58 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 02:49:58 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: <0A29AA96-4626-464E-A912-772A01016367@stufft.io> On Jun 4, 2013, at 2:45 AM, Lennart Regebro wrote: > On Tue, Jun 4, 2013 at 7:46 AM, Donald Stufft wrote: >> Just want to apologize to anyone who recently received a lot of mail from >> me. >> >> I realized shortly after I queued all the emails that they should have been >> grouped by user and not by package. > > I like it this way. Now I can easily reply and include the other > owners in the discussion. :-) > > But on the other hand, I really only need two discussions, one for > zope.* and one for plone.* > > All the packages that I am truly a maintainer for I changed the first > day the feature was available, I don't seem to get emails about those. > So there. :-) > > //Lennart I *was* smart enough to make sure it only emailed for packages that it actually needed too. Just not smart enough not to send people one per package :/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Jun 4 08:53:35 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 02:53:35 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <20130604085235.927888k9i0rzvfeb@mail.tiolive.com> References: <20130604085235.927888k9i0rzvfeb@mail.tiolive.com> Message-ID: <321F68D7-23A4-4182-B4FA-F8CB30795651@stufft.io> On Jun 4, 2013, at 2:52 AM, "Vincent Pelletier" wrote: > As to the technical change: how can this be set when uploading a package for the first time ? > Specifically, is there a control knob available to setup.py (setuptools/distutils/whatever, I'm pretty much lost between the alternatives and usually just cargo-cult when creating a new setup.py) ? > If not, what is/will be the default setting ? (I guess #3 for one month, then #1 maybe ?) > > Regards, > -- > Vincent Pelletier > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > The default option for all new packages since introduced a week or two ago has been #1. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Tue Jun 4 08:45:19 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 4 Jun 2013 08:45:19 +0200 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 7:46 AM, Donald Stufft wrote: > Just want to apologize to anyone who recently received a lot of mail from > me. > > I realized shortly after I queued all the emails that they should have been > grouped by user and not by package. I like it this way. Now I can easily reply and include the other owners in the discussion. :-) But on the other hand, I really only need two discussions, one for zope.* and one for plone.* All the packages that I am truly a maintainer for I changed the first day the feature was available, I don't seem to get emails about those. So there. :-) //Lennart From donald at stufft.io Tue Jun 4 08:54:31 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 02:54:31 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <20130604085235.927888k9i0rzvfeb@mail.tiolive.com> References: <20130604085235.927888k9i0rzvfeb@mail.tiolive.com> Message-ID: On Jun 4, 2013, at 2:52 AM, "Vincent Pelletier" wrote: > As to the technical change: how can this be set when uploading a package for the first time ? > Specifically, is there a control knob available to setup.py (setuptools/distutils/whatever, I'm pretty much lost between the alternatives and usually just cargo-cult when creating a new setup.py) ? > If not, what is/will be the default setting ? (I guess #3 for one month, then #1 maybe ?) > > Regards, > -- > Vincent Pelletier > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > And to answer your other question there's nothing in setup.py to control it. For new packages it defaults to strict and allows folks to, if they really want to, expand it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vincent at nexedi.com Tue Jun 4 08:52:35 2013 From: vincent at nexedi.com (Vincent Pelletier) Date: Tue, 04 Jun 2013 08:52:35 +0200 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: <20130604085235.927888k9i0rzvfeb@mail.tiolive.com> As to the technical change: how can this be set when uploading a package for the first time ? Specifically, is there a control knob available to setup.py (setuptools/distutils/whatever, I'm pretty much lost between the alternatives and usually just cargo-cult when creating a new setup.py) ? If not, what is/will be the default setting ? (I guess #3 for one month, then #1 maybe ?) Regards, -- Vincent Pelletier ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From chris at python.org Tue Jun 4 09:14:24 2013 From: chris at python.org (Chris Withers) Date: Tue, 04 Jun 2013 08:14:24 +0100 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: <51AD893D.1020502@python.org> <51AD8B3B.4070708@python.org> Message-ID: <51AD93D0.6040008@python.org> On 04/06/2013 07:45, Noah Kantrowitz wrote: > >>> As someone also in the top percentile of package maintainers I understand your annoyance, but just make a filter for donald at python.org for the day or something. The vast majority of PyPI users have only one package so asking us to derail the sending (probably resulting in having to begin again) is unhelpful. If the only cost to us all is hitting Ctrl-A Delete, I welcome progress with open arms. >> >> Are you not concerned that various bits of python.org involved in this process are going to start getting hit by RBLs and other spam filtering that will cause problems down the line as a result of all this noise? > > No, because only a very small number of people are going to be getting more than a handful of these, and any user with 50+ packages is hopefully enough of a power-user to not fly off the handle. That's not what I'm referring to; how much mail has actually been sent? MTAs end up being blacklisted automatically by ISPs and RBLs if they heuristically look like they're spewing spam. It's what companies like MailChimp and co spend their lives working around. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From noah at coderanger.net Tue Jun 4 09:16:50 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 4 Jun 2013 00:16:50 -0700 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <51AD93D0.6040008@python.org> References: <51AD893D.1020502@python.org> <51AD8B3B.4070708@python.org> <51AD93D0.6040008@python.org> Message-ID: <707B9B20-7F4F-4DFB-9121-1FF3FB82CE3B@coderanger.net> On Jun 4, 2013, at 12:14 AM, Chris Withers wrote: > On 04/06/2013 07:45, Noah Kantrowitz wrote: >> >>>> As someone also in the top percentile of package maintainers I understand your annoyance, but just make a filter for donald at python.org for the day or something. The vast majority of PyPI users have only one package so asking us to derail the sending (probably resulting in having to begin again) is unhelpful. If the only cost to us all is hitting Ctrl-A Delete, I welcome progress with open arms. >>> >>> Are you not concerned that various bits of python.org involved in this process are going to start getting hit by RBLs and other spam filtering that will cause problems down the line as a result of all this noise? >> >> No, because only a very small number of people are going to be getting more than a handful of these, and any user with 50+ packages is hopefully enough of a power-user to not fly off the handle. > > That's not what I'm referring to; how much mail has actually been sent? > > MTAs end up being blacklisted automatically by ISPs and RBLs if they heuristically look like they're spewing spam. It's what companies like MailChimp and co spend their lives working around. Thanks to running many very very large mailing lists, I can promise you another few thousand messages exiting our servers is a non-issue. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris at python.org Tue Jun 4 09:21:33 2013 From: chris at python.org (Chris Withers) Date: Tue, 04 Jun 2013 08:21:33 +0100 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <707B9B20-7F4F-4DFB-9121-1FF3FB82CE3B@coderanger.net> References: <51AD893D.1020502@python.org> <51AD8B3B.4070708@python.org> <51AD93D0.6040008@python.org> <707B9B20-7F4F-4DFB-9121-1FF3FB82CE3B@coderanger.net> Message-ID: <51AD957D.2090403@python.org> On 04/06/2013 08:16, Noah Kantrowitz wrote: > >> MTAs end up being blacklisted automatically by ISPs and RBLs if they heuristically look like they're spewing spam. It's what companies like MailChimp and co spend their lives working around. > > Thanks to running many very very large mailing lists, I can promise you another few thousand messages exiting our servers is a non-issue. This isn't a mailing list, this is a process sending out mails. I've recently seen something very similar start getting RBL errors from msn after only a hundred or so emails sent. Anyway, let's just hope... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From noah at coderanger.net Tue Jun 4 09:36:39 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 4 Jun 2013 00:36:39 -0700 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <51AD957D.2090403@python.org> References: <51AD893D.1020502@python.org> <51AD8B3B.4070708@python.org> <51AD93D0.6040008@python.org> <707B9B20-7F4F-4DFB-9121-1FF3FB82CE3B@coderanger.net> <51AD957D.2090403@python.org> Message-ID: <3F08839F-6DAD-444B-9B11-AF9E18A62B37@coderanger.net> On Jun 4, 2013, at 12:21 AM, Chris Withers wrote: > On 04/06/2013 08:16, Noah Kantrowitz wrote: >> >>> MTAs end up being blacklisted automatically by ISPs and RBLs if they heuristically look like they're spewing spam. It's what companies like MailChimp and co spend their lives working around. >> >> Thanks to running many very very large mailing lists, I can promise you another few thousand messages exiting our servers is a non-issue. > > This isn't a mailing list, this is a process sending out mails. > > I've recently seen something very similar start getting RBL errors from msn after only a hundred or so emails sent. > > Anyway, let's just hope? Its all the same, these emails are relaying via mail.python.org. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ronaldoussoren at mac.com Tue Jun 4 10:23:22 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 4 Jun 2013 10:23:22 +0200 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: <1E9C2780-07EE-40E9-A41C-2CD845066DDB@mac.com> On 31 May, 2013, at 22:17, Chris Barker - NOAA Federal wrote: > HI Folks, > > A few of us over on the pythonmac list have been hashing out how best > to support binary packages on OS-X. The binary wheel option seems very > promising. > > However, there is one Mac-specific issue that does not seem to be addressed: > > On OS-X, binaries can be "universal" -- what this means is that they > have more than one binary architecture embedded in a single file; the > system selects the right one at run time. Both executables and dynamic > libraries can be universal. In theory, an universal binary can > currently contain up to 4 architectures: This isn't really a problem, distutils uses labels for the set of supported architectures as the architecture label in binary distributions (e.g. pyobjc_core-2.5-py3.4-macosx-10.8-intel.egg for an egg that supports the 'intel' set of architectures (x86_64 and i386), see _osx_support.get_platform_osx in the CPython repository for the labels that are currently used. This is not ideal, but works good enough for now. In the long run this should likely be replaced by the compressed tags sets from PEP 425 (at the cost of a much longer file names). A much more interesting question is which binary wheels should be considered when installing a package, there a two clear ways to select binaries and both have there usecases. Let's say you're running on OSX 10.8 with a CPU that supports x86_64 and a Python installation with support for x86_64, i386 and ppc and compiled with support for OSX 10.4 or later ("MACOSX_DEPLOYMENT_TARGET=10.4"). When installing a package you can pick either: 1) a binary wheel that supports this machine (a wheel compiled with deployment target 10.8 with only x86_64 support) 2) a binary wheel that supports the same systems as the Python installations The former is good enough when you only want to run code on your machine, the latter can be needed when you want to deploy the installed binaries to another machine (for example because you are using py2app to create an application bundle that will be installed elsewhere). > > On my system right now, I have 32bitPPC + 32bit i386 universal python, > running on an intel system: > > In [2]: distutils.util.get_platform() > Out[2]: 'macosx-10.3-i386' On my machine this returns the correct value: 'macosx-10.8-intel' (binary supports 10.8 or later and the i386 and x86_64 architectures). I do know this used to return wrong values on some version of OSX when using the Apple installation of Python because of the way they build Python. Ronald From dsuch at zato.io Tue Jun 4 10:25:05 2013 From: dsuch at zato.io (Dariusz Suchojad) Date: Tue, 04 Jun 2013 10:25:05 +0200 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: On 06/02/2013 05:42 PM, Jason R. Coombs wrote: > Nothing has changed from the 0.7b4 pre-release. This is the version that > will be uploaded to PyPI after we work out the technique to deploy to > PyPI without interfering with the setuptools 0.6 releases. Hi there, congratulations on the release! One thing I'd like to say though is that I released a piece of software that has setuptools pinned to 0.6c12dev-r88846 in buildout and it's not possible to install it now. Coincidentally, the latest release of that software of mine was just several hours before setuptools 0.7 was out so all my tests went fine but in a moment people began to hit upon a wall. Tough luck, well, I would've spotted it right away. There's nothing special about 0.6c12dev-r88846 - it's simply one of ~70 dependencies I pinned down so I could have repeatable builds with buildout. In the future, I'll need to have my own package index, that's for sure. But for now, given that you said you'd like not to interfere with previous releases, could you please upload 0.6c12dev-r88846 back to http://peak.telecommunity.com/snapshots/ or other places where it can be usually found? Many thanks! -- Dariusz Suchojad https://zato.io The next generation ESB and application server. Open-source. In Python. From bojieli at gmail.com Tue Jun 4 11:33:55 2013 From: bojieli at gmail.com (Bojie Li) Date: Tue, 4 Jun 2013 17:33:55 +0800 Subject: [Distutils] Proposal of a new PyPI mirror Message-ID: Hello, I'm from University of Science and Technology of China and have set up a PyPI mirror at http://pypi.mirrors.ustc.edu.cn/ This mirror is synced using bandersnatch every 5 minutes, which is compliant with PEP-381. Can we become an official PyPI mirror? Thanks, -- Linux User Group University of Science and Technology of China Homepage: http://lug.ustc.edu.cn/ E-Mail: lug at ustc.edu.cn Bojie Li From marius at pov.lt Tue Jun 4 12:17:20 2013 From: marius at pov.lt (Marius Gedminas) Date: Tue, 4 Jun 2013 13:17:20 +0300 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <0A29AA96-4626-464E-A912-772A01016367@stufft.io> References: <0A29AA96-4626-464E-A912-772A01016367@stufft.io> Message-ID: <20130604101719.GA29274@fridge.pov.lt> On Tue, Jun 04, 2013 at 02:49:58AM -0400, Donald Stufft wrote: > I *was* smart enough to make sure it only emailed for packages that it > actually needed too. > > Just not smart enough not to send people one per package :/ Heh, for some packages I actually got two emails. I'm guessing I'm listed twice in the roles table. Stephan Richter had a script that granted access to all of zope.* to somebody; perhaps I was manually to a couple of packages before that script ran. Or something. PyPI could use something like GitHub organization accounts. Marius Gedminas -- C is a language that combines all the elegance and power of assembly language with all the readability and maintainability of assembly language. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From donald at stufft.io Tue Jun 4 13:26:16 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 07:26:16 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <20130604101719.GA29274@fridge.pov.lt> References: <0A29AA96-4626-464E-A912-772A01016367@stufft.io> <20130604101719.GA29274@fridge.pov.lt> Message-ID: On Jun 4, 2013, at 6:17 AM, Marius Gedminas wrote: > On Tue, Jun 04, 2013 at 02:49:58AM -0400, Donald Stufft wrote: >> I *was* smart enough to make sure it only emailed for packages that it >> actually needed too. >> >> Just not smart enough not to send people one per package :/ > > Heh, for some packages I actually got two emails. > > I'm guessing I'm listed twice in the roles table. Stephan Richter had > a script that granted access to all of zope.* to somebody; perhaps I > was manually to a couple of packages before that script ran. Or > something. PyPI could use something like GitHub organization accounts. > > Marius Gedminas > -- > C is a language that combines all the elegance and power of assembly language > with all the readability and maintainability of assembly language. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Organization accounts are something I want to add in the future. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Tue Jun 4 13:40:56 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 4 Jun 2013 11:40:56 +0000 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: <0A29AA96-4626-464E-A912-772A01016367@stufft.io> <20130604101719.GA29274@fridge.pov.lt> Message-ID: <20130604114056.GY7789@merlinux.eu> On Tue, Jun 04, 2013 at 07:26 -0400, Donald Stufft wrote: > On Jun 4, 2013, at 6:17 AM, Marius Gedminas wrote: > > > On Tue, Jun 04, 2013 at 02:49:58AM -0400, Donald Stufft wrote: > >> I *was* smart enough to make sure it only emailed for packages that it > >> actually needed too. > >> > >> Just not smart enough not to send people one per package :/ > > > > Heh, for some packages I actually got two emails. > > > > I'm guessing I'm listed twice in the roles table. Stephan Richter had > > a script that granted access to all of zope.* to somebody; perhaps I > > was manually to a couple of packages before that script ran. Or > > something. PyPI could use something like GitHub organization accounts. > > > > Marius Gedminas > > -- > > C is a language that combines all the elegance and power of assembly language > > with all the readability and maintainability of assembly language. > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > Organization accounts are something I want to add in the future. FYI the upcoming (this week) devpi-server release will have organisational github-style accounts. best, holger > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From kencochrane at gmail.com Tue Jun 4 13:43:18 2013 From: kencochrane at gmail.com (ken cochrane) Date: Tue, 4 Jun 2013 07:43:18 -0400 Subject: [Distutils] b.pypi.python.org Message-ID: b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. Two things: 1. Now that we have the pypi CDN up, do we still need this mirror? 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. Thank you, Ken Cochrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jun 4 14:01:26 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 08:01:26 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: On Jun 4, 2013, at 7:43 AM, ken cochrane wrote: > b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine > > b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. > > I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. > > I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. > I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. > > I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. > > Two things: > > 1. Now that we have the pypi CDN up, do we still need this mirror? Honestly probably not. Mirrors are less important from a availability/speed side of things now and will likely move to being more useful for companies and such to use. > > 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. I don't have such permission sadly. > > Thank you, > Ken Cochrane > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jim at zope.com Tue Jun 4 15:28:48 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 4 Jun 2013 09:28:48 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 1:46 AM, Donald Stufft wrote: > Just want to apologize to anyone who recently received a lot of mail from > me. > > I realized shortly after I queued all the emails that they should have been > grouped by user and not by package. > > So, I'm sorry. I really am and I feel bad about the amount of mail some of > you have received from me. This is a minor annoyance. If someone is worked up about this, they aren't using the right email client, or don't know how to use it. I was able to deal with these messages in a few seconds. Much thanks for all your packaging efforts. I really appreciate it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From kencochrane at gmail.com Tue Jun 4 16:33:38 2013 From: kencochrane at gmail.com (ken cochrane) Date: Tue, 4 Jun 2013 10:33:38 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: > > > b.pypi.python.org is an official mirror that runs on Google App engine, > and it uses a special mirror package built just for GAE. Code for it is > found here. https://bitbucket.org/loewis/pypi-appengine > > b.pypi.python.org has been broken for over 104 days according to > http://www.pypi-mirrors.org, and this is because of an issue when we > switched pypi over to serving over SSL. > > I have submitted a pull request to fix this. > https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. > > I am one of the maintainers of b.pypi.python.org, so I can see the logs > and push out a new version. > I haven't needed to push a version out before, and I'm a little hesitant > incase I do it wrong and break something. > > I also don't want to push code to GAE from my fork, until my PR gets > accepted or else someone else in the future might deploy the original one > again and remove my fix. > > Two things: > > 1. Now that we have the pypi CDN up, do we still need this mirror? > > > Honestly probably not. Mirrors are less important from a > availability/speed side of things now and will likely move to being more > useful for companies and such to use. > OK, what would be the procedure for removing a mirror? Anyone know who is in charge of this mirror? I think Guido had it setup when he worked at Google, and google is paying for the costs of the mirror, but now that he doesn't work for Google, not sure who might be the contact person on that side. If we can get it up to date again, I think it is fine, but an out of date mirror is not useful to anyone, and it could cause problems in the long run. > > 2. If yes to 1. if someone can take a minute to review my PR, and leave > comments, or if you have the power, accept my pull request and push out a > new version so we can get the mirror up to date. > > > I don't have such permission sadly. > > Thank you anyway. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jun 4 16:35:10 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 10:35:10 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: <8D158897-D247-404A-9AED-F197BF64C645@stufft.io> On Jun 4, 2013, at 10:33 AM, ken cochrane wrote: > >> b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine >> >> b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. >> >> I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >> >> I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. >> I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. >> >> I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. >> >> Two things: >> >> 1. Now that we have the pypi CDN up, do we still need this mirror? > > Honestly probably not. Mirrors are less important from a availability/speed side of things now and will likely move to being more useful for companies and such to use. > > OK, what would be the procedure for removing a mirror? Anyone know who is in charge of this mirror? I think Guido had it setup when he worked at Google, and google is paying for the costs of the mirror, but now that he doesn't work for Google, not sure who might be the contact person on that side. Procedure is probably someone decide to remove it and ask here, and then we route b.pypi.python.org back to PyPI so the address doesn't stop working. > > If we can get it up to date again, I think it is fine, but an out of date mirror is not useful to anyone, and it could cause problems in the long run. > >> >> 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. > > I don't have such permission sadly. > > > Thank you anyway. > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Tue Jun 4 16:59:33 2013 From: brett at python.org (Brett Cannon) Date: Tue, 4 Jun 2013 10:59:33 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 10:33 AM, ken cochrane wrote: > >> b.pypi.python.org is an official mirror that runs on Google App engine, >> and it uses a special mirror package built just for GAE. Code for it is >> found here. https://bitbucket.org/loewis/pypi-appengine >> >> b.pypi.python.org has been broken for over 104 days according to >> http://www.pypi-mirrors.org, and this is because of an issue when we >> switched pypi over to serving over SSL. >> >> I have submitted a pull request to fix this. >> https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >> >> I am one of the maintainers of b.pypi.python.org, so I can see the logs >> and push out a new version. >> I haven't needed to push a version out before, and I'm a little hesitant >> incase I do it wrong and break something. >> >> I also don't want to push code to GAE from my fork, until my PR gets >> accepted or else someone else in the future might deploy the original one >> again and remove my fix. >> >> Two things: >> >> 1. Now that we have the pypi CDN up, do we still need this mirror? >> >> >> Honestly probably not. Mirrors are less important from a >> availability/speed side of things now and will likely move to being more >> useful for companies and such to use. >> > > OK, what would be the procedure for removing a mirror? Anyone know who is > in charge of this mirror? I think Guido had it setup when he worked at > Google, and google is paying for the costs of the mirror, but now that he > doesn't work for Google, not sure who might be the contact person on that > side. > I've asked Guido who are admins on the account it to get it turned off. If he doesn't know I can try to find out internally. -Brett > > If we can get it up to date again, I think it is fine, but an out of date > mirror is not useful to anyone, and it could cause problems in the long run. > > >> >> 2. If yes to 1. if someone can take a minute to review my PR, and leave >> comments, or if you have the power, accept my pull request and push out a >> new version so we can get the mirror up to date. >> >> >> I don't have such permission sadly. >> >> > Thank you anyway. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Tue Jun 4 17:11:33 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 04 Jun 2013 11:11:33 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/04/2013 01:46 AM, Donald Stufft wrote: > Just want to apologize to anyone who recently received a lot of mail > from me. > > I realized shortly after I queued all the emails that they should > have been grouped by user and not by package. > > So, I'm sorry. I really am and I feel bad about the amount of mail > some of you have received from me. I don't mind the 250 or so messages, but I'd *really* like to get a "bulk change" UI to set all 250 to #1 at once -- there is no way I'm going to take the time to clickety-click through them and fix them by hand. Without such a UI, I guess I'm just going to wait for the auto-conversion. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGuA6UACgkQ+gerLs4ltQ5CuACeMst3VewOCkGMx82Dx62PTtan Bj0AnRpSCCymJGMdxjuVx0/klKqIYxfo =IELQ -----END PGP SIGNATURE----- From donald at stufft.io Tue Jun 4 17:13:06 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 11:13:06 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: References: Message-ID: <8FB92031-3518-4D4A-BDA2-0AF99CD6449F@stufft.io> On Jun 4, 2013, at 11:11 AM, Tres Seaver wrote: > Signed PGP part > On 06/04/2013 01:46 AM, Donald Stufft wrote: > > Just want to apologize to anyone who recently received a lot of mail > > from me. > > > > I realized shortly after I queued all the emails that they should > > have been grouped by user and not by package. > > > > So, I'm sorry. I really am and I feel bad about the amount of mail > > some of you have received from me. > > I don't mind the 250 or so messages, but I'd *really* like to get a "bulk > change" UI to set all 250 to #1 at once -- there is no way I'm going to > take the time to clickety-click through them and fix them by hand. > Without such a UI, I guess I'm just going to wait for the auto-conversion. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I'll see what I can whip up. Would a script be ok? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tseaver at palladion.com Tue Jun 4 17:18:27 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 04 Jun 2013 11:18:27 -0400 Subject: [Distutils] Preemptive Apology for Volume of Mail In-Reply-To: <8FB92031-3518-4D4A-BDA2-0AF99CD6449F@stufft.io> References: <8FB92031-3518-4D4A-BDA2-0AF99CD6449F@stufft.io> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/04/2013 11:13 AM, Donald Stufft wrote: > I'll see what I can whip up. Would a script be ok? Sure -- I'm fine with whatever works. Thanks! Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGuBUMACgkQ+gerLs4ltQ4+QACeKSfBL2qVzMtphRX+JZ3cmNPQ 0XcAniXUgAIlLaH729gO3GItgUzD0kMW =NVwu -----END PGP SIGNATURE----- From kencochrane at gmail.com Tue Jun 4 17:22:20 2013 From: kencochrane at gmail.com (ken cochrane) Date: Tue, 4 Jun 2013 11:22:20 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: > b.pypi.python.org is an official mirror that runs on Google App engine, >>> and it uses a special mirror package built just for GAE. Code for it is >>> found here. https://bitbucket.org/loewis/pypi-appengine >>> >>> b.pypi.python.org has been broken for over 104 days according to >>> http://www.pypi-mirrors.org, and this is because of an issue when we >>> switched pypi over to serving over SSL. >>> >>> I have submitted a pull request to fix this. >>> https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>> >>> I am one of the maintainers of b.pypi.python.org, so I can see the logs >>> and push out a new version. >>> I haven't needed to push a version out before, and I'm a little hesitant >>> incase I do it wrong and break something. >>> >>> I also don't want to push code to GAE from my fork, until my PR gets >>> accepted or else someone else in the future might deploy the original one >>> again and remove my fix. >>> >>> Two things: >>> >>> 1. Now that we have the pypi CDN up, do we still need this mirror? >>> >>> >>> Honestly probably not. Mirrors are less important from a >>> availability/speed side of things now and will likely move to being more >>> useful for companies and such to use. >>> >> >> OK, what would be the procedure for removing a mirror? Anyone know who is >> in charge of this mirror? I think Guido had it setup when he worked at >> Google, and google is paying for the costs of the mirror, but now that he >> doesn't work for Google, not sure who might be the contact person on that >> side. >> > > I've asked Guido who are admins on the account it to get it turned off. If > he doesn't know I can try to find out internally. > > Brett, Here is the owner list that I can see on GAE - guido - kencochrane (me) - kumar.mcmillan - martin.v.loewis - r1chardj0n3s Ken > >> If we can get it up to date again, I think it is fine, but an out of date >> mirror is not useful to anyone, and it could cause problems in the long run. >> >> >>> >>> 2. If yes to 1. if someone can take a minute to review my PR, and leave >>> comments, or if you have the power, accept my pull request and push out a >>> new version so we can get the mirror up to date. >>> >>> >>> I don't have such permission sadly. >>> >>> >> Thank you anyway. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Jun 4 18:27:10 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 4 Jun 2013 09:27:10 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: <1E9C2780-07EE-40E9-A41C-2CD845066DDB@mac.com> References: <1E9C2780-07EE-40E9-A41C-2CD845066DDB@mac.com> Message-ID: On Tue, Jun 4, 2013 at 1:23 AM, Ronald Oussoren wrote: > This isn't really a problem, distutils uses labels for the set of supported architectures as the architecture label in binary distributions (e.g. pyobjc_core-2.5-py3.4-macosx-10.8-intel.egg for an egg that supports the 'intel' set of architectures (x86_64 and i386), see _osx_support.get_platform_osx in the CPython repository for the labels that are currently used. > > This is not ideal, but works good enough for now. In the long run this should likely be replaced by the compressed tags sets from PEP 425 (at the cost of a much longer file names). I think now may be the "long run". But in any case, as we were discussing, as in theory there could be lots of possible combinations, in practice there are only two in the wild (plus the non-universal ones...), and probably only one we really need to keep supporting, so simply have a canonical name for the one or two is fine. > A much more interesting question is which binary wheels should be considered when installing a package, indeed. > there a two clear ways to select binaries and both have there usecases. Let's say you're running on OSX 10.8 with a CPU that supports x86_64 and a Python installation with support for x86_64, i386 and ppc and compiled with support for OSX 10.4 or later ("MACOSX_DEPLOYMENT_TARGET=10.4"). When installing a package you can pick either: > > 1) a binary wheel that supports this machine (a wheel compiled with deployment target 10.8 with only x86_64 support) can you link a newer deployment target-built lib with an older deployment-target executable? I guess so... > 2) a binary wheel that supports the same systems as the Python installations > The former is good enough when you only want to run code on your machine, the latter can be needed when you want to deploy the installed binaries to another machine (for example because you are using py2app to create an application bundle that will be installed elsewhere). I would say that we should clearly _prefer_ a wheel that fully supports the pyton the installer is used with. But I'm a bit ambivalent about what to do if there is no such wheel found, but there is one that supports the currently running architecture. I"m inclined to say don't install it: - We build the universal binaries to support people that don't want to know or care about such details. - We want to support universal binary wheels for those folks. - The primary reason to build binary wheels at all is to support non-native architectures -- it's a whole lot easier to simply build for your current machine, and homebrew and macports, and ??? support that already. All that being said, I sometimes think that we should simply have a 32 bit and 64 bit binary out there, and forget all this universal stuff! It works OK for Windows.... >> On my system right now, I have 32bitPPC + 32bit i386 universal python, >> running on an intel system: >> >> In [2]: distutils.util.get_platform() >> Out[2]: 'macosx-10.3-i386' > On my machine this returns the correct value: 'macosx-10.8-intel' (binary supports 10.8 or later and the i386 and x86_64 architectures). I do know this used to return wrong values on some version of OSX when using the Apple installation of Python because of the way they build Python. This isn't an Apple build -- I'm pretty sure it's the one from Python.org...though looking now it's only 2.7.2 -- I guess I haven't upgraded in a while! now with 2.7.5: In [3]: distutils.util.get_platform() Out[3]: 'macosx-10.3-fat' I guess "fat" means 32bit ppc + intel, deployment target 10.3 ? So we may want to update distutils.util.get_platform to do a PEP 425 name, but we've got something that could work now. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From chris.barker at noaa.gov Tue Jun 4 18:35:20 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 4 Jun 2013 09:35:20 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: On Mon, Jun 3, 2013 at 3:47 PM, Daniel Holth wrote: > That would make sense. Can you come up with code to detect that a > newly compiled extension is universal, and that a Python is? It looks like distutils.util.get_platform() now does the right thing for knowing what the currently running pyton is (see Ronald's message) For determining the status of a newly compiled extension, I usually simple run the "file" command line utility on it: $ file python python: Mach-O universal binary with 2 architectures python (for architecture ppc): Mach-O executable ppc python (for architecture i386): Mach-O executable i386 (works for *.so, too...) that could easily be parsed out, but we would still want to know the deployment target, which maybe you could get by parsing otool output: $ otool -L python python (architecture ppc): /Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0) /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 47.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.10) python (architecture i386): /Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.10) (compatibility version of libSystem???) But I imagine there is a cleaner way -- Ronald?? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ronaldoussoren at mac.com Tue Jun 4 18:55:06 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 4 Jun 2013 18:55:06 +0200 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: On 4 Jun, 2013, at 18:35, Chris Barker - NOAA Federal wrote: > On Mon, Jun 3, 2013 at 3:47 PM, Daniel Holth wrote: > >> That would make sense. Can you come up with code to detect that a >> newly compiled extension is universal, and that a Python is? > > It looks like distutils.util.get_platform() now does the right thing > for knowing what the currently running pyton is (see Ronald's message) Correct. Barring installation errors, such as with some older releases of OSX, distutils.util.get_platform() already returns the correct information (both the supported architectures and the deployment target). > > For determining the status of a newly compiled extension, I usually > simple run the "file" command line utility on it: > > $ file python > python: Mach-O universal binary with 2 architectures > python (for architecture ppc): Mach-O executable ppc > python (for architecture i386): Mach-O executable i386 > > (works for *.so, too...) > > that could easily be parsed out, but we would still want to know the > deployment target, which maybe you could get by parsing otool output: > > $ otool -L python > python (architecture ppc): > /Library/Frameworks/Python.framework/Versions/2.7/Python > (compatibility version 2.7.0, current version 2.7.0) > /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 47.1.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 88.3.10) > python (architecture i386): > /Library/Frameworks/Python.framework/Versions/2.7/Python > (compatibility version 2.7.0, current version 2.7.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 88.3.10) > > (compatibility version of libSystem???) > > But I imagine there is a cleaner way -- Ronald?? The output of 'otool -l' lists (amongst others) the value of LC_VERSION_MIN_MACOSX which is the deployment target. Both can also be found using macholib ;-) Ronald From dholth at gmail.com Tue Jun 4 18:59:06 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 4 Jun 2013 12:59:06 -0400 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 12:55 PM, Ronald Oussoren wrote: > > On 4 Jun, 2013, at 18:35, Chris Barker - NOAA Federal wrote: > >> On Mon, Jun 3, 2013 at 3:47 PM, Daniel Holth wrote: >> >>> That would make sense. Can you come up with code to detect that a >>> newly compiled extension is universal, and that a Python is? >> >> It looks like distutils.util.get_platform() now does the right thing >> for knowing what the currently running pyton is (see Ronald's message) > > Correct. Barring installation errors, such as with some older releases of OSX, distutils.util.get_platform() already returns the correct information (both the supported architectures and the deployment target). >> >> For determining the status of a newly compiled extension, I usually >> simple run the "file" command line utility on it: >> >> $ file python >> python: Mach-O universal binary with 2 architectures >> python (for architecture ppc): Mach-O executable ppc >> python (for architecture i386): Mach-O executable i386 >> >> (works for *.so, too...) >> >> that could easily be parsed out, but we would still want to know the >> deployment target, which maybe you could get by parsing otool output: >> >> $ otool -L python >> python (architecture ppc): >> /Library/Frameworks/Python.framework/Versions/2.7/Python >> (compatibility version 2.7.0, current version 2.7.0) >> /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 47.1.0) >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >> version 88.3.10) >> python (architecture i386): >> /Library/Frameworks/Python.framework/Versions/2.7/Python >> (compatibility version 2.7.0, current version 2.7.0) >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >> version 88.3.10) >> >> (compatibility version of libSystem???) >> >> But I imagine there is a cleaner way -- Ronald?? > > The output of 'otool -l' lists (amongst others) the value of LC_VERSION_MIN_MACOSX which is the deployment target. > > Both can also be found using macholib ;-) > > Ronald FYI pip (the development version that supports wheel) uses https://github.com/pypa/pip/blob/develop/pip/pep425tags.py to discover which tags it will consider. The tags are listed in order of most- to least-preferred; a file with a tag nearer the beginning of the list will be preferred when there is more than one choice for a particular version of a distribution. From ronaldoussoren at mac.com Tue Jun 4 19:06:04 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 4 Jun 2013 19:06:04 +0200 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: <1E9C2780-07EE-40E9-A41C-2CD845066DDB@mac.com> Message-ID: On 4 Jun, 2013, at 18:27, Chris Barker - NOAA Federal wrote: > On Tue, Jun 4, 2013 at 1:23 AM, Ronald Oussoren wrote: > >> This isn't really a problem, distutils uses labels for the set of supported architectures as the architecture label in binary distributions (e.g. pyobjc_core-2.5-py3.4-macosx-10.8-intel.egg for an egg that supports the 'intel' set of architectures (x86_64 and i386), see _osx_support.get_platform_osx in the CPython repository for the labels that are currently used. >> >> This is not ideal, but works good enough for now. In the long run this should likely be replaced by the compressed tags sets from PEP 425 (at the cost of a much longer file names). > > I think now may be the "long run". Maybe, at least for 3.4 (assuming that wheel support gets in the stdlib by then). > But in any case, as we were > discussing, as in theory there could be lots of possible combinations, > in practice there are only two in the wild (plus the non-universal > ones...), and probably only one we really need to keep supporting, so > simply have a canonical name for the one or two is fine. > >> A much more interesting question is which binary wheels should be considered when installing a package, > > indeed. > >> there a two clear ways to select binaries and both have there usecases. Let's say you're running on OSX 10.8 with a CPU that supports x86_64 and a Python installation with support for x86_64, i386 and ppc and compiled with support for OSX 10.4 or later ("MACOSX_DEPLOYMENT_TARGET=10.4"). When installing a package you can pick either: >> >> 1) a binary wheel that supports this machine (a wheel compiled with deployment target 10.8 with only x86_64 support) > > can you link a newer deployment target-built lib with an older > deployment-target executable? I guess so... Yes. I've used extensions with a newer deployment target than the python executable, and even the other way around should work. > >> 2) a binary wheel that supports the same systems as the Python installations > >> The former is good enough when you only want to run code on your machine, the latter can be needed when you want to deploy the installed binaries to another machine (for example because you are using py2app to create an application bundle that will be installed elsewhere). > > I would say that we should clearly _prefer_ a wheel that fully > supports the pyton the installer is used with. > > But I'm a bit ambivalent about what to do if there is no such wheel > found, but there is one that supports the currently running > architecture. I"m inclined to say don't install it: I'd currently prefer to only install binaries that exactly match the python installation, that's the most conservative solution, requires no changes to installation tools and is easier to reason about. > > - We build the universal binaries to support people that don't want > to know or care about such details. > > - We want to support universal binary wheels for those folks. > > - The primary reason to build binary wheels at all is to support > non-native architectures -- it's a whole lot easier to simply build > for your current machine, and homebrew and macports, and ??? support > that already. > > All that being said, I sometimes think that we should simply have a 32 > bit and 64 bit binary out there, and forget all this universal stuff! > It works OK for Windows.... For some definition of OK ;-), the registry can be fun when you're running 32 binaries on a 64-bit windows. A clear advantage of universal binaries is that you can have one set of binaries and don't have to pick which architecture gets the most obvious installation directory (e.g. /lib vs. /lib64 on a lot of Linux systems). > >>> On my system right now, I have 32bitPPC + 32bit i386 universal python, >>> running on an intel system: >>> >>> In [2]: distutils.util.get_platform() >>> Out[2]: 'macosx-10.3-i386' > >> On my machine this returns the correct value: 'macosx-10.8-intel' (binary supports 10.8 or later and the i386 and x86_64 architectures). I do know this used to return wrong values on some version of OSX when using the Apple installation of Python because of the way they build Python. > > This isn't an Apple build -- I'm pretty sure it's the one from > Python.org...though looking now it's only 2.7.2 -- I guess I haven't > upgraded in a while! > > now with 2.7.5: > > In [3]: distutils.util.get_platform() > Out[3]: 'macosx-10.3-fat' > > I guess "fat" means 32bit ppc + intel, deployment target 10.3 ? That's correct. > > So we may want to update distutils.util.get_platform to do a PEP 425 > name, but we've got something that could work now. That can wait until the new packaging stuff (wheels, metadata 2.0, distlib, ...) gets merged, and even then it might be nicer to keep 'fat' and 'intel' as short names. Ronald From pje at telecommunity.com Tue Jun 4 23:13:31 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 4 Jun 2013 17:13:31 -0400 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: On Tue, Jun 4, 2013 at 4:25 AM, Dariusz Suchojad wrote: > But for now, given that you said you'd like not to interfere with previous > releases, could you please upload 0.6c12dev-r88846 back to > http://peak.telecommunity.com/snapshots/ or other places where it can be > usually found? The snapshots are generated automatically from SVN and are not kept around: old ones are automatically deleted. So, even if I were to check out and old version and build it, it wouldn't stay up for very long. I wouldn't recommend pinning a specific version like that except via reference to an svn URL. From dsuch at zato.io Tue Jun 4 23:40:46 2013 From: dsuch at zato.io (Dariusz Suchojad) Date: Tue, 04 Jun 2013 23:40:46 +0200 Subject: [Distutils] Setuptools 0.7 final available for download In-Reply-To: References: <6b854b06b790448f88461bca6234406f@BY2PR06MB011.namprd06.prod.outlook.com> Message-ID: <51AE5EDE.7000809@zato.io> > The snapshots are generated automatically from SVN and are not kept > around: old ones are automatically deleted. So, even if I were to > check out and old version and build it, it wouldn't stay up for very > long. OK, I see. Can you please add a note to that effect somewhere on PyPI here https://pypi.python.org/pypi/setuptools ? 0.6c12dev-r88846 must've been there for at least 6 months because this is when I frozen the version and I didn't do it because I needed this particular one - it got pulled automatically and probably was the latest one at that time. I didn't actually realize old versions would be deleted at all. many thanks again, -- Dariusz Suchojad https://zato.io The next generation ESB and application server. Open-source. In Python. From chris.barker at noaa.gov Tue Jun 4 23:39:53 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 4 Jun 2013 14:39:53 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: <1E9C2780-07EE-40E9-A41C-2CD845066DDB@mac.com> Message-ID: On Tue, Jun 4, 2013 at 10:06 AM, Ronald Oussoren wrote: >>> This is not ideal, but works good enough for now. In the long run this should likely be replaced by the compressed tags sets from PEP 425 (at the cost of a much longer file names). >> >> I think now may be the "long run". > > Maybe, at least for 3.4 (assuming that wheel support gets in the stdlib by then). yup. > I'd currently prefer to only install binaries that exactly match the python installation, that's the most conservative solution, requires no changes to installation tools and is easier to reason about. we've got a consensus of two now -- good enough for me ;-) >> All that being said, I sometimes think that we should simply have a 32 >> bit and 64 bit binary out there, and forget all this universal stuff! >> It works OK for Windows.... > > For some definition of OK ;-), the registry can be fun when you're running 32 binaries on a 64-bit windows. A clear advantage of universal binaries is that you can have one set of binaries and don't have to pick which architecture gets the most obvious installation directory (e.g. /lib vs. /lib64 on a lot of Linux systems). yeah -- I do like them, when they work... >> So we may want to update distutils.util.get_platform to do a PEP 425 >> name, but we've got something that could work now. > That can wait until the new packaging stuff (wheels, metadata 2.0, distlib, ...) gets merged, and even then it might be nicer to keep 'fat' and 'intel' as short names. it would be best for any change to get merged around the same time as the packaging stuff -- so there is only one versin out there. But I agree, the short names are nice -- if the only place anyone would get the names is distutils, then we're fine. This is all closer that I thought was -- nice! -Chris > > Ronald > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From chris.barker at noaa.gov Tue Jun 4 23:53:44 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 4 Jun 2013 14:53:44 -0700 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 9:55 AM, Ronald Oussoren wrote: >> $ otool -L python >> python (architecture ppc): >> /Library/Frameworks/Python.framework/Versions/2.7/Python >> (compatibility version 2.7.0, current version 2.7.0) >> /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 47.1.0) >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >> version 88.3.10) >> python (architecture i386): >> /Library/Frameworks/Python.framework/Versions/2.7/Python >> (compatibility version 2.7.0, current version 2.7.0) >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >> version 88.3.10) >> >> (compatibility version of libSystem???) >> >> But I imagine there is a cleaner way -- Ronald?? > > The output of 'otool -l' lists (amongst others) the value of LC_VERSION_MIN_MACOSX which is the deployment target. I don't see that on my machine -- is that "l", a lower-case "el'? > Both can also be found using macholib ;-) probably best not to add another dependency. But I've lot track on why we'd need this -- presumably the wheel builder would know what it built, and could supply the proper meta-data, and that would be that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From barry at python.org Wed Jun 5 00:16:16 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 4 Jun 2013 18:16:16 -0400 Subject: [Distutils] option #1 plus download_url scraping Message-ID: <20130604181616.576d1773@anarchist> Like many of you, I got Donald's message about the changes to URLs for Cheeseshop packages. My question is about the three options; I think I want a middle ground, but I'm interested to see why you will discourage me from that . IIUC, option #1 is fine for packages hosted on PyPI. But what if our packages are *also* hosted elsewhere, say for redundancy purposes, and that external location needs to be scraped? Specifically, say I have a download_url in my setup.py. I *want* that url to be essentially a wildcard or index page because I don't want to have to change setup.py every time I make a release (unless of course `setup.py sdist` did it for me). I also can't add this url to the "Additional File URLs" page for my package because again I'd have to change it every time I do a release. So the middle ground I think I want is: option #1 plus scraping from download_url, but only download_url. Am I a horrible person for wanting this? Is there a better way. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From noah at coderanger.net Wed Jun 5 00:23:03 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 4 Jun 2013 15:23:03 -0700 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130604181616.576d1773@anarchist> References: <20130604181616.576d1773@anarchist> Message-ID: <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> On Jun 4, 2013, at 3:16 PM, Barry Warsaw wrote: > Like many of you, I got Donald's message about the changes to URLs for > Cheeseshop packages. My question is about the three options; I think I want a > middle ground, but I'm interested to see why you will discourage me from that > . > > IIUC, option #1 is fine for packages hosted on PyPI. But what if our packages > are *also* hosted elsewhere, say for redundancy purposes, and that external > location needs to be scraped? > > Specifically, say I have a download_url in my setup.py. I *want* that url to > be essentially a wildcard or index page because I don't want to have to change > setup.py every time I make a release (unless of course `setup.py sdist` did it > for me). I also can't add this url to the "Additional File URLs" page for my > package because again I'd have to change it every time I do a release. > > So the middle ground I think I want is: option #1 plus scraping from > download_url, but only download_url. > > Am I a horrible person for wanting this? Is there a better way. Do you mean you just don't want to update the version number in setup.py before you release? I'm a bit unsure of the reason for this. The goal is very specifically the hosting outside of PyPI is no longer encouraged. The reliability and performance of PyPI have enough of a track record now that "I want it on my own site just in case" no longer holds enough water to be worth the substantial downsides. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Jun 5 00:30:54 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 18:30:54 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130604181616.576d1773@anarchist> References: <20130604181616.576d1773@anarchist> Message-ID: I'm interested in the use case. How are generating a release without running setup.py sdist? On Jun 4, 2013, at 6:16 PM, Barry Warsaw wrote: > Like many of you, I got Donald's message about the changes to URLs for > Cheeseshop packages. My question is about the three options; I think I want a > middle ground, but I'm interested to see why you will discourage me from that > . > > IIUC, option #1 is fine for packages hosted on PyPI. But what if our packages > are *also* hosted elsewhere, say for redundancy purposes, and that external > location needs to be scraped? > > Specifically, say I have a download_url in my setup.py. I *want* that url to > be essentially a wildcard or index page because I don't want to have to change > setup.py every time I make a release (unless of course `setup.py sdist` did it > for me). I also can't add this url to the "Additional File URLs" page for my > package because again I'd have to change it every time I do a release. > > So the middle ground I think I want is: option #1 plus scraping from > download_url, but only download_url. > > Am I a horrible person for wanting this? Is there a better way. > > Cheers, > -Barry > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From carl at oddbird.net Wed Jun 5 00:46:24 2013 From: carl at oddbird.net (Carl Meyer) Date: Tue, 04 Jun 2013 16:46:24 -0600 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: References: <20130604181616.576d1773@anarchist> Message-ID: <51AE6E40.5040009@oddbird.net> On 06/04/2013 04:30 PM, Donald Stufft wrote: > I'm interested in the use case. How are generating a release without running setup.py sdist? I think you misunderstood (the way Barry described it wasn't completely clear). If I'm reading it correctly, "I don't want to have to change setup.py every time I make a release" means "I don't want to have to change the Download-URL metadata in setup.py to point to a specific new sdist tarball URL every time I make a release; I want it to always point to the same index URL and have that index page scraped for new release tarball URLs." Carl From carl at oddbird.net Wed Jun 5 00:43:24 2013 From: carl at oddbird.net (Carl Meyer) Date: Tue, 04 Jun 2013 16:43:24 -0600 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130604181616.576d1773@anarchist> References: <20130604181616.576d1773@anarchist> Message-ID: <51AE6D8C.9030004@oddbird.net> Hi Barry, On 06/04/2013 04:16 PM, Barry Warsaw wrote: > Like many of you, I got Donald's message about the changes to URLs for > Cheeseshop packages. My question is about the three options; I think I want a > middle ground, but I'm interested to see why you will discourage me from that > . > > IIUC, option #1 is fine for packages hosted on PyPI. But what if our packages > are *also* hosted elsewhere, say for redundancy purposes, and that external > location needs to be scraped? > > Specifically, say I have a download_url in my setup.py. I *want* that url to > be essentially a wildcard or index page because I don't want to have to change > setup.py every time I make a release (unless of course `setup.py sdist` did it > for me). I also can't add this url to the "Additional File URLs" page for my > package because again I'd have to change it every time I do a release. > > So the middle ground I think I want is: option #1 plus scraping from > download_url, but only download_url. > > Am I a horrible person for wanting this? Is there a better way. The first question, of course, is "why not just host on PyPI"? If "redundancy" is the real reason, you might think about whether that reason still applies with the new PyPI infrastructure, CDN, etc. But let's presume that whatever your reason for hosting off-PyPI, it's a good one. (To be clear, PEP 438 takes the position that there are and will continue to be some good reasons, and the option of off-PyPI hosting - in some form - should be supported indefinitely). The problem with the current system is that client installer tools do the scraping whenever they search for installation candidates for your package, which means that you are asking every user of your package to accept an unnecessary slowdown every single time they install. But the information on that download_url page should only change when you make a release, so the scraping should really be done just once, at release time, and the resulting sdist URL(s) stored on PyPI so that installers can take them into account without fetching or scraping any additional pages. So the idea is that to satisfy your use-case, there should be a tool that you can use at release-time to scrape your downloads page and automatically add sdist URLs found there to the explicit URLs list on PyPI. That tool, of course, doesn't exist yet :-) Until someone builds it, you'll have to stay with option #3 (and accept that you are slowing down installations for your users) to satisfy your use case. Carl From holger at merlinux.eu Wed Jun 5 00:48:30 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 4 Jun 2013 22:48:30 +0000 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> Message-ID: <20130604224830.GB7789@merlinux.eu> On Tue, Jun 04, 2013 at 15:23 -0700, Noah Kantrowitz wrote: > > On Jun 4, 2013, at 3:16 PM, Barry Warsaw wrote: > > > Like many of you, I got Donald's message about the changes to URLs for > > Cheeseshop packages. My question is about the three options; I think I want a > > middle ground, but I'm interested to see why you will discourage me from that > > . > > > > IIUC, option #1 is fine for packages hosted on PyPI. But what if our packages > > are *also* hosted elsewhere, say for redundancy purposes, and that external > > location needs to be scraped? > > > > Specifically, say I have a download_url in my setup.py. I *want* that url to > > be essentially a wildcard or index page because I don't want to have to change > > setup.py every time I make a release (unless of course `setup.py sdist` did it > > for me). I also can't add this url to the "Additional File URLs" page for my > > package because again I'd have to change it every time I do a release. > > > > So the middle ground I think I want is: option #1 plus scraping from > > download_url, but only download_url. > > > > Am I a horrible person for wanting this? Is there a better way. > > Do you mean you just don't want to update the version number in setup.py before you release? I'm a bit unsure of the reason for this. The goal is very specifically the hosting outside of PyPI is no longer encouraged. I agree with not encouraging "hosting outside" but i'd like to sidenote that PEP438 has a section discussing history/reasons for external hosting [1] and concludes with this: Irrespective of the present-day validity of these reasons, there clearly is a history why people choose to host files externally and it even was for some time the only way you could do things. This PEP takes the position that there remain some valid reasons for external hosting even today. As to Barry's need, what might be missing is a tool that helps to register links with a checksum. This would not require a change to setup.py but it would require an extra (scriptable) action when releasing a package. OTOH, just automating a bit of setup.py changes might be easier and would allow hosting on pypi which is usually more reliable for install time users. best, holger > The reliability and performance of PyPI have enough of a track record now that "I want it on my own site just in case" no longer holds enough water to be worth the substantial downsides. > > --Noah > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Wed Jun 5 01:15:16 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 19:15:16 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130604181616.576d1773@anarchist> References: <20130604181616.576d1773@anarchist> Message-ID: <7F1E6256-E4AA-435F-941C-5BF158185A86@stufft.io> On Jun 4, 2013, at 6:16 PM, Barry Warsaw wrote: > Like many of you, I got Donald's message about the changes to URLs for > Cheeseshop packages. My question is about the three options; I think I want a > middle ground, but I'm interested to see why you will discourage me from that > . > > IIUC, option #1 is fine for packages hosted on PyPI. But what if our packages > are *also* hosted elsewhere, say for redundancy purposes, and that external > location needs to be scraped? > > Specifically, say I have a download_url in my setup.py. I *want* that url to > be essentially a wildcard or index page because I don't want to have to change > setup.py every time I make a release (unless of course `setup.py sdist` did it > for me). I also can't add this url to the "Additional File URLs" page for my > package because again I'd have to change it every time I do a release. > > So the middle ground I think I want is: option #1 plus scraping from > download_url, but only download_url. > > Am I a horrible person for wanting this? Is there a better way. > > Cheers, > -Barry > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I was originally on my phone and am now back at the computer, so I can give a longer reply now. So my first question is as to your actual use case, what are you attempting to achieve by hosting externally instead of on PyPI? It's likely there's a better way but what problem are you actually attempting to solve :) You mention reliability but as far as I can tell it's basically impossible to add more reliability to the system via external urls. The only method an installation client has to discover your external urls is PyPI, and if PyPI is up to enable them to discover them, then it should also be up to enable them to download directly from PyPI. Additionally except in one specific circumstance it's also a major security issue. Installers can download the /simple/ pages via verified TLS, and then use that to verify the hashes on that page to verify the download file. When you're scraping an external page the only time that is *safe* to do is if that page is a) served via verified TLS and b) has a supported hash fragment for every single file an installer might attempt to download. Furthermore the scraping adds an extreme amount of time to the installation. I recently did basically what pip does sans downloading the actual packages across all of PyPI. So I processed every /simple/ page, looked on it for other pages to scrape, downloaded and scraped those. That process took about 3 days to complete. If I run the same process but simulating a world where everyone was using #1, the process takes about 10 minutes to complete. The PEP concludes that there are valid reasons to host externally, but I'm of the opinion if there is a valid reason, it is an extreme edge case and likely would be better solved another way. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Wed Jun 5 05:32:40 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 5 Jun 2013 05:32:40 +0200 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130604181616.576d1773@anarchist> References: <20130604181616.576d1773@anarchist> Message-ID: On Wed, Jun 5, 2013 at 12:16 AM, Barry Warsaw wrote: > But what if our packages > are *also* hosted elsewhere, say for redundancy purposes, and that external > location needs to be scraped? If PyPI is down, you wont find that link, so it won't be scraped anyhow. //Lennart From ronaldoussoren at mac.com Wed Jun 5 07:56:39 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 5 Jun 2013 07:56:39 +0200 Subject: [Distutils] Binary Wheels and "universal" builds on OS-X In-Reply-To: References: Message-ID: <016929D4-9309-4106-99AF-1C1D9AE8DA42@mac.com> On 4 Jun, 2013, at 23:53, Chris Barker - NOAA Federal wrote: > On Tue, Jun 4, 2013 at 9:55 AM, Ronald Oussoren wrote: > >>> $ otool -L python >>> python (architecture ppc): >>> /Library/Frameworks/Python.framework/Versions/2.7/Python >>> (compatibility version 2.7.0, current version 2.7.0) >>> /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 47.1.0) >>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >>> version 88.3.10) >>> python (architecture i386): >>> /Library/Frameworks/Python.framework/Versions/2.7/Python >>> (compatibility version 2.7.0, current version 2.7.0) >>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >>> version 88.3.10) >>> >>> (compatibility version of libSystem???) >>> >>> But I imagine there is a cleaner way -- Ronald?? >> >> The output of 'otool -l' lists (amongst others) the value of LC_VERSION_MIN_MACOSX which is the deployment target. > > I don't see that on my machine -- is that "l", a lower-case "el'? The option or the LC_VERSION_MIN_MACOSX value? I'm running OSX 10.8, where both are available, but according to Google LC_VERSION_MIN_MACOSX was introduced in a 10.6 subrelease. Without that load command you cannot easily see what the deployment target is (the best way I find while writing would be a tool that checks library load commands and the list of symbols used with a to-be-maintained registry). > >> Both can also be found using macholib ;-) > > probably best not to add another dependency. > > But I've lot track on why we'd need this -- presumably the wheel > builder would know what it built, and could supply the proper > meta-data, and that would be that. You don't need this, distutils (and hence setuptools) already knows this information and other build tools can also know this. Ronald > > -Chris > > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov From maphew at gmail.com Wed Jun 5 08:19:15 2013 From: maphew at gmail.com (Matt Wilkie) Date: Tue, 4 Jun 2013 23:19:15 -0700 Subject: [Distutils] problems with sdist upload since CDN update In-Reply-To: <619B8DAE-8716-430B-8053-D28A83CE4ED7@stufft.io> References: <51A76B55.5030207@cryptelium.net> <7F3F8244-8940-4D30-9FC2-236297581A55@stufft.io> <2AC2FE75-B60F-4AB9-92A1-F50001AC6039@stufft.io> <619B8DAE-8716-430B-8053-D28A83CE4ED7@stufft.io> Message-ID: Hi Donald, > I'm guessing that you didn't have a long_description when it wasn't working, > and your README file was named README.TXT, and between when it wasn't > working and when it was you added a long_description. Thanks for investigating this. There is supposed to be a long description, but it could be that it is not generated as expected or is not working in some way. I'm not a very good programmer yet :) Ahh, I think I've got it. Setup.py generates long_description from README.TXT if present, or from a hard coded boiler plate if not. In preparation for a new release coming up soon, another team member deleted all content in the Readme, since it was outdated and misleading. So I presume PyPi saw the empty long_description and bailed. -matt From barry at python.org Wed Jun 5 16:25:54 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 5 Jun 2013 10:25:54 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> Message-ID: <20130605102554.71e6bf6d@anarchist> On Jun 04, 2013, at 03:23 PM, Noah Kantrowitz wrote: >Do you mean you just don't want to update the version number in setup.py >before you release? Correct, although on further reflection, if the alternative download site has predictable URLs, then it wouldn't be too difficult to calculate the new url for setup.py. I just don't want to have to change the version number in more than one place (which for my packages, is *not* the setup.py). >I'm a bit unsure of the reason for this. The goal is very specifically the >hosting outside of PyPI is no longer encouraged. The reliability and >performance of PyPI have enough of a track record now that "I want it on my >own site just in case" no longer holds enough water to be worth the >substantial downsides. Fair enough. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Jun 5 17:04:47 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 5 Jun 2013 11:04:47 -0400 Subject: [Distutils] option #1 plus download_url scraping References: <20130604181616.576d1773@anarchist> <51AE6E40.5040009@oddbird.net> Message-ID: <20130605110447.2ae68be1@anarchist> On Jun 04, 2013, at 04:46 PM, Carl Meyer wrote: >On 06/04/2013 04:30 PM, Donald Stufft wrote: >> I'm interested in the use case. How are generating a release without >> running setup.py sdist? > >I think you misunderstood (the way Barry described it wasn't completely >clear). If I'm reading it correctly, "I don't want to have to change >setup.py every time I make a release" means "I don't want to have to >change the Download-URL metadata in setup.py to point to a specific new >sdist tarball URL every time I make a release; I want it to always point >to the same index URL and have that index page scraped for new release >tarball URLs." Correct, sorry for not being clear. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From donald at stufft.io Wed Jun 5 18:16:09 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 5 Jun 2013 12:16:09 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130605102554.71e6bf6d@anarchist> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> Message-ID: On Jun 5, 2013, at 10:25 AM, Barry Warsaw wrote: > On Jun 04, 2013, at 03:23 PM, Noah Kantrowitz wrote: > >> Do you mean you just don't want to update the version number in setup.py >> before you release? > > Correct, although on further reflection, if the alternative download site has > predictable URLs, then it wouldn't be too difficult to calculate the new url > for setup.py. I just don't want to have to change the version number in more > than one place (which for my packages, is *not* the setup.py). > >> I'm a bit unsure of the reason for this. The goal is very specifically the >> hosting outside of PyPI is no longer encouraged. The reliability and >> performance of PyPI have enough of a track record now that "I want it on my >> own site just in case" no longer holds enough water to be worth the >> substantial downsides. > > Fair enough. > > -Barry > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Where are you updating the version information at? And how are you generating a tarball so that it's name has the correct version in it? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Wed Jun 5 19:49:48 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 5 Jun 2013 13:49:48 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> Message-ID: <20130605134948.3e7374f6@anarchist> On Jun 05, 2013, at 12:16 PM, Donald Stufft wrote: >Where are you updating the version information at? And how are you generating >a tarball so that it's name has the correct version in it? It depends on the package, but let's say it's in a version.txt file. Your implication is correct though - if setup.py is parsing that file to calculate the version key, it can also do the same and calculate the download_url value. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From donald at stufft.io Wed Jun 5 20:47:58 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 5 Jun 2013 14:47:58 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130605134948.3e7374f6@anarchist> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> <20130605134948.3e7374f6@anarchist> Message-ID: <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> On Jun 5, 2013, at 1:49 PM, Barry Warsaw wrote: > On Jun 05, 2013, at 12:16 PM, Donald Stufft wrote: > >> Where are you updating the version information at? And how are you generating >> a tarball so that it's name has the correct version in it? > > It depends on the package, but let's say it's in a version.txt file. Your > implication is correct though - if setup.py is parsing that file to calculate > the version key, it can also do the same and calculate the download_url value. > > -Barry I'm really just trying to get a sense of your workflow to see if I can make any changes to improve the process for it. One of the big problems with download_url is that the data in setup.py is used in (and influences the content of) the final dist file. This means that inside of a setup.py you won't know what the hash of the final file is. So it's difficult for a setup.py based workflow with external urls to provide md5 sums for the files which means that pip and friends can't verify that no body modified the download in transit. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Wed Jun 5 21:11:14 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 5 Jun 2013 15:11:14 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> <20130605134948.3e7374f6@anarchist> <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> Message-ID: <20130605151114.202987fd@anarchist> On Jun 05, 2013, at 02:47 PM, Donald Stufft wrote: >I'm really just trying to get a sense of your workflow to see if I can make >any changes to improve the process for it. > >One of the big problems with download_url is that the data in setup.py is >used in (and influences the content of) the final dist file. This means that >inside of a setup.py you won't know what the hash of the final file is. So >it's difficult for a setup.py based workflow with external urls to provide >md5 sums for the files which means that pip and friends can't verify that no >body modified the download in transit. Let me explain what I (used to) do, and I'll let you decide whether anything needs to change. ;) When I've finally got my vcs into a releasable state, I'll generally do: $ python setup.py sdist upload -s As you know, this will create the tarball and signature file in dist, and upload everything nicely to the Cheeseshop. At this point, I go to my project's Launchpad page and push the big "I made a release" button. This fiddles some state on my project page, and it allows me to upload files attached to that particular release. The nice thing is that I can just upload the dist/*.tar.gz and dist/*.asc to add the tarball and signature to the Launchpad download page. E.g. https://launchpad.net/flufl.enum and https://launchpad.net/flufl.enum/+download The url is predictable (which is good because it also has to play nicely with Debian watch files), with option #3, I just added the index page to download_url and let clients scrape it. You'll see that it contains links to the md5 checksum and the locally generated signature. There must be some value to also allowing folks to download from Launchpad, as shown by the 1055 downloads of flufl.enum. Where are the PyPI download stats? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From donald at stufft.io Wed Jun 5 21:35:09 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 5 Jun 2013 15:35:09 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <20130605151114.202987fd@anarchist> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> <20130605134948.3e7374f6@anarchist> <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> <20130605151114.202987fd@anarchist> Message-ID: <7EEAC60E-9DB8-4EA0-91DF-0E50B669A1AD@stufft.io> On Jun 5, 2013, at 3:11 PM, Barry Warsaw wrote: > On Jun 05, 2013, at 02:47 PM, Donald Stufft wrote: > >> I'm really just trying to get a sense of your workflow to see if I can make >> any changes to improve the process for it. >> >> One of the big problems with download_url is that the data in setup.py is >> used in (and influences the content of) the final dist file. This means that >> inside of a setup.py you won't know what the hash of the final file is. So >> it's difficult for a setup.py based workflow with external urls to provide >> md5 sums for the files which means that pip and friends can't verify that no >> body modified the download in transit. > > Let me explain what I (used to) do, and I'll let you decide whether anything > needs to change. ;) > > When I've finally got my vcs into a releasable state, I'll generally do: > > $ python setup.py sdist upload -s > > As you know, this will create the tarball and signature file in dist, and > upload everything nicely to the Cheeseshop. At this point, I go to my > project's Launchpad page and push the big "I made a release" button. This > fiddles some state on my project page, and it allows me to upload files > attached to that particular release. The nice thing is that I can just upload > the dist/*.tar.gz and dist/*.asc to add the tarball and signature to the > Launchpad download page. E.g. > > https://launchpad.net/flufl.enum > > and > > https://launchpad.net/flufl.enum/+download > > The url is predictable (which is good because it also has to play nicely with > Debian watch files), with option #3, I just added the index page to > download_url and let clients scrape it. You'll see that it contains links to > the md5 checksum and the locally generated signature. > > There must be some value to also allowing folks to download from Launchpad, as > shown by the 1055 downloads of flufl.enum. Where are the PyPI download stats? > > -Barry > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Ah ok! I understand what you're trying to do now, thanks :) Right now download counts are disabled on PyPI due to some issues with the script that integrates them pegging the CPU, and then the CDN. But prior to that flufl.enum had 28196 downloads from PyPI (total across all versions). So Launchpad doesn't provide the md5 sums in a way that the tools will be able to process them, however you actually got lucky in that both your download url, and the files themselves are available via verifiable SSL so they aren't insecure if someone is using pip 1.3+ (and maybe newer easy_install? not sure the state of SSL outside of pip). I think the downloads you see are either people manually downloading it, or tools that don't prefer PyPI hosted urls that just happened to pick the launchpad url. I think for this the best option is to just continue uploading everything to PyPI and switch to #1 (which I think I saw you did). While launchpad is verifiable via SSL, and is unlikely to have bad uptime I don't think it provides any benefit for the folks installing your package so there's not much of a reason to keep it around on your /simple/ page. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Thu Jun 6 00:08:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Jun 2013 08:08:46 +1000 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> <20130605134948.3e7374f6@anarchist> <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> Message-ID: On 6 Jun 2013 04:49, "Donald Stufft" wrote: > > > On Jun 5, 2013, at 1:49 PM, Barry Warsaw wrote: > >> On Jun 05, 2013, at 12:16 PM, Donald Stufft wrote: >> >>> Where are you updating the version information at? And how are you generating >>> a tarball so that it's name has the correct version in it? >> >> >> It depends on the package, but let's say it's in a version.txt file. Your >> implication is correct though - if setup.py is parsing that file to calculate >> the version key, it can also do the same and calculate the download_url value. >> >> -Barry > > > I'm really just trying to get a sense of your workflow to see if I can make any changes to improve the process for it. > > One of the big problems with download_url is that the data in setup.py is used in (and influences the content of) the final dist file. This means that inside of a setup.py you won't know what the hash of the final file is. So it's difficult for a setup.py based workflow with external urls to provide md5 sums for the files which means that pip and friends can't verify that no body modified the download in transit. Hmm, I should mention this problem in PEP 426, and explicitly limit source_url to tarballs and VCS references. This self-referencing problem means it can't easily refer to a built sdist anyway, and the original source is preferred for distro packaging purposes. Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu Jun 6 00:52:36 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 5 Jun 2013 18:52:36 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> <20130605134948.3e7374f6@anarchist> <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> Message-ID: On Wed, Jun 5, 2013 at 2:47 PM, Donald Stufft wrote: > One of the big problems with download_url is that the data in setup.py is > used in (and influences the content of) the final dist file. This means that > inside of a setup.py you won't know what the hash of the final file is. So > it's difficult for a setup.py based workflow with external urls to provide > md5 sums for the files which means that pip and friends can't verify that no > body modified the download in transit. Not if it's done in a setup.py command that runs after the distributions are built, akin to the way the upload command works now. If there were, say, an "uplink" command based on a modified version of upload, it could call the PyPI API to pass along hashed URLs. At some point I intend to write such a command so that my current snapshot scripts (which run on the server the downloads are hosted from) can update PyPI with properly hashed URLs. (But I'm not sure when "some point" will be, exactly, so if someone else writes it first I'll be a happy camper.) From donald at stufft.io Thu Jun 6 00:56:01 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 5 Jun 2013 18:56:01 -0400 Subject: [Distutils] option #1 plus download_url scraping In-Reply-To: References: <20130604181616.576d1773@anarchist> <03BD9A02-89F0-4794-9681-ECB115773BF5@coderanger.net> <20130605102554.71e6bf6d@anarchist> <20130605134948.3e7374f6@anarchist> <806C45DD-7891-4B76-91A1-6A60ED67C022@stufft.io> Message-ID: On Jun 5, 2013, at 6:52 PM, PJ Eby wrote: > On Wed, Jun 5, 2013 at 2:47 PM, Donald Stufft wrote: >> One of the big problems with download_url is that the data in setup.py is >> used in (and influences the content of) the final dist file. This means that >> inside of a setup.py you won't know what the hash of the final file is. So >> it's difficult for a setup.py based workflow with external urls to provide >> md5 sums for the files which means that pip and friends can't verify that no >> body modified the download in transit. > > Not if it's done in a setup.py command that runs after the > distributions are built, akin to the way the upload command works now. > If there were, say, an "uplink" command based on a modified version > of upload, it could call the PyPI API to pass along hashed URLs. > > At some point I intend to write such a command so that my current > snapshot scripts (which run on the server the downloads are hosted > from) can update PyPI with properly hashed URLs. (But I'm not sure > when "some point" will be, exactly, so if someone else writes it first > I'll be a happy camper.) With static metadata ideally PyPI will be reading metadata from inside of the uploaded file and all that will be required is for publishing tools to push the file up. However something like your uplink command would (assuming I understand it correctly) work fine because those "additional urls to list on the /simple/ page" are not part of the package metadata. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From richard at python.org Thu Jun 6 06:13:16 2013 From: richard at python.org (Richard Jones) Date: Thu, 6 Jun 2013 14:13:16 +1000 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: On 5 June 2013 01:22, ken cochrane wrote: > Brett, > > Here is the owner list that I can see on GAE > > - guido > - kencochrane (me) > - kumar.mcmillan > - martin.v.loewis > - r1chardj0n3s > orly? I wouldn't know what to do with that :-) [if there is something I can do then I'm happy to!] -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 6 08:29:32 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 6 Jun 2013 02:29:32 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: On Jun 6, 2013, at 12:13 AM, Richard Jones wrote: > On 5 June 2013 01:22, ken cochrane wrote: > Brett, > > Here is the owner list that I can see on GAE > > - guido > - kencochrane (me) > - kumar.mcmillan > - martin.v.loewis > - r1chardj0n3s > > orly? I wouldn't know what to do with that :-) [if there is something I can do then I'm happy to!] > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig FWIW I fully support killing b.pypi.python.org. It's been a constant problem with being months out of date all the time. It seems as soon as it gets fixed within a month or two it's broken again, and stays broken for 30, 60, or 90+ days at a time. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Thu Jun 6 17:26:07 2013 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jun 2013 11:26:07 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 11:22 AM, ken cochrane wrote: > > b.pypi.python.org is an official mirror that runs on Google App engine, >>>> and it uses a special mirror package built just for GAE. Code for it is >>>> found here. https://bitbucket.org/loewis/pypi-appengine >>>> >>>> b.pypi.python.org has been broken for over 104 days according to >>>> http://www.pypi-mirrors.org, and this is because of an issue when we >>>> switched pypi over to serving over SSL. >>>> >>>> I have submitted a pull request to fix this. >>>> https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>>> >>>> I am one of the maintainers of b.pypi.python.org, so I can see the >>>> logs and push out a new version. >>>> I haven't needed to push a version out before, and I'm a little >>>> hesitant incase I do it wrong and break something. >>>> >>>> I also don't want to push code to GAE from my fork, until my PR gets >>>> accepted or else someone else in the future might deploy the original one >>>> again and remove my fix. >>>> >>>> Two things: >>>> >>>> 1. Now that we have the pypi CDN up, do we still need this mirror? >>>> >>>> >>>> Honestly probably not. Mirrors are less important from a >>>> availability/speed side of things now and will likely move to being more >>>> useful for companies and such to use. >>>> >>> >>> OK, what would be the procedure for removing a mirror? Anyone know who >>> is in charge of this mirror? I think Guido had it setup when he worked >>> at Google, and google is paying for the costs of the mirror, but now that >>> he doesn't work for Google, not sure who might be the contact person on >>> that side. >>> >> >> I've asked Guido who are admins on the account it to get it turned off. >> If he doesn't know I can try to find out internally. >> >> > Brett, > > Here is the owner list that I can see on GAE > > - guido > - kencochrane (me) > - kumar.mcmillan > - martin.v.loewis > - r1chardj0n3s > So Guido didn't even know he was an owner. =) In terms of shutting down the app, you will want to do two things. First is empty out the cron.yaml file; it should have nothing more than "cron:". After that you probably want to return 404 for everything; see http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. If you want, Ken, I can clone https://bitbucket.org/kencochrane/pypi-appengine and send you a pull request to do all of this since I recently did something similar for py3ksupport.appspot.com when I shut it down. -Brett > > Ken > > >> >>> If we can get it up to date again, I think it is fine, but an out of >>> date mirror is not useful to anyone, and it could cause problems in the >>> long run. >>> >>> >>>> >>>> 2. If yes to 1. if someone can take a minute to review my PR, and leave >>>> comments, or if you have the power, accept my pull request and push out a >>>> new version so we can get the mirror up to date. >>>> >>>> >>>> I don't have such permission sadly. >>>> >>>> >>> Thank you anyway. >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Thu Jun 6 21:31:09 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 6 Jun 2013 19:31:09 +0000 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: References: <20130530182820.GT7789@merlinux.eu> Message-ID: <20130606193109.GG7789@merlinux.eu> On Thu, May 30, 2013 at 22:26 -0400, Daniel Holth wrote: > On Thu, May 30, 2013 at 2:28 PM, holger krekel wrote: > No one actually parses PKG-INFO so I expect the Metadata 2.0 > transition to be relatively painless. I am actually intending to do exactly that in devpi-server. It's the only method to obtain metadata from a tar file without running setup.py ASFAIK. That being said, when a new format and file comes around i am happy to detect that instead, i guess. holger From dholth at gmail.com Thu Jun 6 22:41:08 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 6 Jun 2013 16:41:08 -0400 Subject: [Distutils] Draft PEP for JSON based metadata published In-Reply-To: <20130606193109.GG7789@merlinux.eu> References: <20130530182820.GT7789@merlinux.eu> <20130606193109.GG7789@merlinux.eu> Message-ID: On Thu, Jun 6, 2013 at 3:31 PM, holger krekel wrote: > On Thu, May 30, 2013 at 22:26 -0400, Daniel Holth wrote: >> On Thu, May 30, 2013 at 2:28 PM, holger krekel wrote: >> No one actually parses PKG-INFO so I expect the Metadata 2.0 >> transition to be relatively painless. > > I am actually intending to do exactly that in devpi-server. > It's the only method to obtain metadata from a tar file without > running setup.py ASFAIK. > > That being said, when a new format and file comes around > i am happy to detect that instead, i guess. Unfortunately if you author a pypi server I count you as "no one" for this exercise. :-P And the most useful requirements metadata is in a setuptools .txt file anyway. Once you start seeing sdists with a .dist-info directory instead of an .egg-info or nothing you'll just look for pymeta.json. Wheel has code to convert from .egg-info to the newer formats. From ubershmekel at gmail.com Fri Jun 7 07:59:48 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Fri, 7 Jun 2013 08:59:48 +0300 Subject: [Distutils] wxPython and wx Message-ID: I wanted to install wx and tried to "sudo pip install wx". Sadly this https://pypi.python.org/pypi/wx module described as "my first python module, like php file_get_contents" shadowed the wxpython I eventually installed with "sudo apt-get install python-wxgtk2.8". Can we remove that package from pypi? Preferably with the author's consent. Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Fri Jun 7 09:27:28 2013 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 7 Jun 2013 09:27:28 +0200 Subject: [Distutils] wxPython and wx In-Reply-To: References: Message-ID: On Fri, Jun 7, 2013 at 7:59 AM, Yuval Greenfield wrote: > I wanted to install wx and tried to "sudo pip install wx". Sadly this > https://pypi.python.org/pypi/wx module described as "my first python module, > like php file_get_contents" shadowed the wxpython I eventually installed > with "sudo apt-get install python-wxgtk2.8". > > Can we remove that package from pypi? Preferably with the author's consent. Packages are allowed to have the same module names as other packages. In this case it's a bit unfortunate, but it should reasonably have been quickly solved by uninstalling the wx package? //Lennart From regebro at gmail.com Fri Jun 7 10:47:17 2013 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 7 Jun 2013 10:47:17 +0200 Subject: [Distutils] wxPython and wx In-Reply-To: References: Message-ID: On Fri, Jun 7, 2013 at 9:56 AM, Yuval Greenfield wrote: > On Fri, Jun 7, 2013 at 10:27 AM, Lennart Regebro wrote: >> >> Packages are allowed to have the same module names as other packages. >> In this case it's a bit unfortunate, but it should reasonably have >> been quickly solved by uninstalling the wx package? >> >> //Lennart > > It wasn't easy figuring out I needed to type "sudo pip uninstall wx" to fix > wx. A python beginner would have had a terrible time with this. I expect a beginner would have read steps on how to install wxPython and hence not installed wx by mistake. :-) The fundamental problem here is of course that wxPython has chosen to have different names for package and module, which is an unfortunate choice. If the wxPython maintainers want to rename the package wx, that would solve the problem. > I realize we don't want to start squat-wars. But pypi's wx is a 5 liner that > absolutely no person would ever want installed. At the least, pip should > warn me when I install "wx" that it's probably not what I'm looking for. There's no way for pip to know that. //Lennart From ubershmekel at gmail.com Fri Jun 7 09:56:25 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Fri, 7 Jun 2013 10:56:25 +0300 Subject: [Distutils] wxPython and wx In-Reply-To: References: Message-ID: On Fri, Jun 7, 2013 at 10:27 AM, Lennart Regebro wrote: > Packages are allowed to have the same module names as other packages. > In this case it's a bit unfortunate, but it should reasonably have > been quickly solved by uninstalling the wx package? > > //Lennart > It wasn't easy figuring out I needed to type "sudo pip uninstall wx" to fix wx. A python beginner would have had a terrible time with this. I realize we don't want to start squat-wars. But pypi's wx is a 5 liner that absolutely no person would ever want installed. At the least, pip should warn me when I install "wx" that it's probably not what I'm looking for. Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Jun 7 17:38:33 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 7 Jun 2013 08:38:33 -0700 Subject: [Distutils] wxPython and wx In-Reply-To: References: Message-ID: On Fri, Jun 7, 2013 at 12:56 AM, Yuval Greenfield wrote: > I realize we don't want to start squat-wars. But pypi's wx is a 5 liner that > absolutely no person would ever want installed. At the least, pip should > warn me when I install "wx" that it's probably not what I'm looking for. This isn't a question for the wxPython project, it's one for the PyPi folks and/or the author of the "other" wx. As it happens, there has been a discussion recently on the distutils list about whether the Pyton community should establish a system for regulating top-level package names -- but at this point, it seems everyone all getting along is where we are at. wxpython is listed on PyPi, what else can we do? This is not a unique situation that someone would accidentally install a different package than they intended. In this case, I suspect that if you installed wxPython on top of that wx, it would just work. Still not a bad idea to suggest the author of the other wx to remove or rename it -- why invite confusion? Last point -- does PyPi have a system for aliases? If so, it might be a good idea to have wxPython in there as wx as well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kencochrane at gmail.com Fri Jun 7 23:26:09 2013 From: kencochrane at gmail.com (ken cochrane) Date: Fri, 7 Jun 2013 17:26:09 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: > >> b.pypi.python.org is an official mirror that runs on Google App >>>>> engine, and it uses a special mirror package built just for GAE. Code for >>>>> it is found here. https://bitbucket.org/loewis/pypi-appengine >>>>> >>>>> b.pypi.python.org has been broken for over 104 days according to >>>>> http://www.pypi-mirrors.org, and this is because of an issue when we >>>>> switched pypi over to serving over SSL. >>>>> >>>>> I have submitted a pull request to fix this. >>>>> https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>>>> >>>>> I am one of the maintainers of b.pypi.python.org, so I can see the >>>>> logs and push out a new version. >>>>> I haven't needed to push a version out before, and I'm a little >>>>> hesitant incase I do it wrong and break something. >>>>> >>>>> I also don't want to push code to GAE from my fork, until my PR gets >>>>> accepted or else someone else in the future might deploy the original one >>>>> again and remove my fix. >>>>> >>>>> Two things: >>>>> >>>>> 1. Now that we have the pypi CDN up, do we still need this mirror? >>>>> >>>>> >>>>> Honestly probably not. Mirrors are less important from a >>>>> availability/speed side of things now and will likely move to being more >>>>> useful for companies and such to use. >>>>> >>>> >>>> OK, what would be the procedure for removing a mirror? Anyone know who >>>> is in charge of this mirror? I think Guido had it setup when he worked >>>> at Google, and google is paying for the costs of the mirror, but now that >>>> he doesn't work for Google, not sure who might be the contact person on >>>> that side. >>>> >>> >>> I've asked Guido who are admins on the account it to get it turned off. >>> If he doesn't know I can try to find out internally. >>> >>> >> Brett, >> >> Here is the owner list that I can see on GAE >> >> - guido >> - kencochrane (me) >> - kumar.mcmillan >> - martin.v.loewis >> - r1chardj0n3s >> > > So Guido didn't even know he was an owner. =) > > In terms of shutting down the app, you will want to do two things. First > is empty out the cron.yaml file; it should have nothing more than "cron:". > After that you probably want to return 404 for everything; see > http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. > > If you want, Ken, I can clone > https://bitbucket.org/kencochrane/pypi-appengine and send you a pull > request to do all of this since I recently did something similar for > py3ksupport.appspot.com when I shut it down. > Brett, If our goal is to shut it down, then yes please, if you can send a pull request that would be great.. We should also probably remove it from the pypi mirror pool before we do this so it no longer gets traffic sent it's way. >> >>> >>>> If we can get it up to date again, I think it is fine, but an out of >>>> date mirror is not useful to anyone, and it could cause problems in the >>>> long run. >>>> >>>> >>>>> >>>>> 2. If yes to 1. if someone can take a minute to review my PR, and >>>>> leave comments, or if you have the power, accept my pull request and push >>>>> out a new version so we can get the mirror up to date. >>>>> >>>>> >>>>> I don't have such permission sadly. >>>>> >>>>> >>>> Thank you anyway. >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jun 7 23:27:48 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 7 Jun 2013 17:27:48 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: <39538F60-678B-42CB-B17D-B62B58409C07@stufft.io> On Jun 7, 2013, at 5:26 PM, ken cochrane wrote: > > >> b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine >> >> b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. >> >> I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >> >> I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. >> I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. >> >> I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. >> >> Two things: >> >> 1. Now that we have the pypi CDN up, do we still need this mirror? > > Honestly probably not. Mirrors are less important from a availability/speed side of things now and will likely move to being more useful for companies and such to use. > > OK, what would be the procedure for removing a mirror? Anyone know who is in charge of this mirror? I think Guido had it setup when he worked at Google, and google is paying for the costs of the mirror, but now that he doesn't work for Google, not sure who might be the contact person on that side. > > I've asked Guido who are admins on the account it to get it turned off. If he doesn't know I can try to find out internally. > > > Brett, > > Here is the owner list that I can see on GAE > > - guido > - kencochrane (me) > - kumar.mcmillan > - martin.v.loewis > - r1chardj0n3s > > So Guido didn't even know he was an owner. =) > > In terms of shutting down the app, you will want to do two things. First is empty out the cron.yaml file; it should have nothing more than "cron:". After that you probably want to return 404 for everything; see http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. > > If you want, Ken, I can clone https://bitbucket.org/kencochrane/pypi-appengine and send you a pull request to do all of this since I recently did something similar for py3ksupport.appspot.com when I shut it down. > > Brett, > > If our goal is to shut it down, then yes please, if you can send a pull request that would be great.. We should also probably remove it from the pypi mirror pool before we do this so it no longer gets traffic sent it's way. > > > > > > If we can get it up to date again, I think it is fine, but an out of date mirror is not useful to anyone, and it could cause problems in the long run. > >> >> 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. > > I don't have such permission sadly. > > > Thank you anyway. > > > > > Paging Noah to kill the DNS Unless there's any objections to removing it? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Fri Jun 7 23:34:40 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Fri, 7 Jun 2013 14:34:40 -0700 Subject: [Distutils] b.pypi.python.org In-Reply-To: <39538F60-678B-42CB-B17D-B62B58409C07@stufft.io> References: <39538F60-678B-42CB-B17D-B62B58409C07@stufft.io> Message-ID: On Jun 7, 2013, at 2:27 PM, Donald Stufft wrote: > > On Jun 7, 2013, at 5:26 PM, ken cochrane wrote: > >> >> >>> b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine >>> >>> b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. >>> >>> I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>> >>> I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. >>> I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. >>> >>> I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. >>> >>> Two things: >>> >>> 1. Now that we have the pypi CDN up, do we still need this mirror? >> >> Honestly probably not. Mirrors are less important from a availability/speed side of things now and will likely move to being more useful for companies and such to use. >> >> OK, what would be the procedure for removing a mirror? Anyone know who is in charge of this mirror? I think Guido had it setup when he worked at Google, and google is paying for the costs of the mirror, but now that he doesn't work for Google, not sure who might be the contact person on that side. >> >> I've asked Guido who are admins on the account it to get it turned off. If he doesn't know I can try to find out internally. >> >> >> Brett, >> >> Here is the owner list that I can see on GAE >> >> - guido >> - kencochrane (me) >> - kumar.mcmillan >> - martin.v.loewis >> - r1chardj0n3s >> >> So Guido didn't even know he was an owner. =) >> >> In terms of shutting down the app, you will want to do two things. First is empty out the cron.yaml file; it should have nothing more than "cron:". After that you probably want to return 404 for everything; see http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. >> >> If you want, Ken, I can clone https://bitbucket.org/kencochrane/pypi-appengine and send you a pull request to do all of this since I recently did something similar for py3ksupport.appspot.com when I shut it down. >> >> Brett, >> >> If our goal is to shut it down, then yes please, if you can send a pull request that would be great.. We should also probably remove it from the pypi mirror pool before we do this so it no longer gets traffic sent it's way. >> >> >> >> >> >> If we can get it up to date again, I think it is fine, but an out of date mirror is not useful to anyone, and it could cause problems in the long run. >> >>> >>> 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. >> >> I don't have such permission sadly. >> >> >> Thank you anyway. >> >> >> >> >> > > Paging Noah to kill the DNS Unless there's any objections to removing it? If no one complains in the next 24-ish hours I'll just point it back at pypi.python.org like we did with d.pypi. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Sat Jun 8 03:56:37 2013 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jun 2013 21:56:37 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: I might not get to it for a week, but I will get to it. On Fri, Jun 7, 2013 at 5:26 PM, ken cochrane wrote: > > >>> b.pypi.python.org is an official mirror that runs on Google App >>>>>> engine, and it uses a special mirror package built just for GAE. Code for >>>>>> it is found here. https://bitbucket.org/loewis/pypi-appengine >>>>>> >>>>>> b.pypi.python.org has been broken for over 104 days according to >>>>>> http://www.pypi-mirrors.org, and this is because of an issue when we >>>>>> switched pypi over to serving over SSL. >>>>>> >>>>>> I have submitted a pull request to fix this. >>>>>> https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>>>>> >>>>>> I am one of the maintainers of b.pypi.python.org, so I can see the >>>>>> logs and push out a new version. >>>>>> I haven't needed to push a version out before, and I'm a little >>>>>> hesitant incase I do it wrong and break something. >>>>>> >>>>>> I also don't want to push code to GAE from my fork, until my PR gets >>>>>> accepted or else someone else in the future might deploy the original one >>>>>> again and remove my fix. >>>>>> >>>>>> Two things: >>>>>> >>>>>> 1. Now that we have the pypi CDN up, do we still need this mirror? >>>>>> >>>>>> >>>>>> Honestly probably not. Mirrors are less important from a >>>>>> availability/speed side of things now and will likely move to being more >>>>>> useful for companies and such to use. >>>>>> >>>>> >>>>> OK, what would be the procedure for removing a mirror? Anyone know who >>>>> is in charge of this mirror? I think Guido had it setup when he >>>>> worked at Google, and google is paying for the costs of the mirror, but >>>>> now that he doesn't work for Google, not sure who might be the contact >>>>> person on that side. >>>>> >>>> >>>> I've asked Guido who are admins on the account it to get it turned off. >>>> If he doesn't know I can try to find out internally. >>>> >>>> >>> Brett, >>> >>> Here is the owner list that I can see on GAE >>> >>> - guido >>> - kencochrane (me) >>> - kumar.mcmillan >>> - martin.v.loewis >>> - r1chardj0n3s >>> >> >> So Guido didn't even know he was an owner. =) >> >> In terms of shutting down the app, you will want to do two things. First >> is empty out the cron.yaml file; it should have nothing more than "cron:". >> After that you probably want to return 404 for everything; see >> http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. >> >> If you want, Ken, I can clone >> https://bitbucket.org/kencochrane/pypi-appengine and send you a pull >> request to do all of this since I recently did something similar for >> py3ksupport.appspot.com when I shut it down. >> > > Brett, > > If our goal is to shut it down, then yes please, if you can send a pull > request that would be great.. We should also probably remove it from the > pypi mirror pool before we do this so it no longer gets traffic sent it's > way. > > > > >>> >>>> >>>>> If we can get it up to date again, I think it is fine, but an out of >>>>> date mirror is not useful to anyone, and it could cause problems in the >>>>> long run. >>>>> >>>>> >>>>>> >>>>>> 2. If yes to 1. if someone can take a minute to review my PR, and >>>>>> leave comments, or if you have the power, accept my pull request and push >>>>>> out a new version so we can get the mirror up to date. >>>>>> >>>>>> >>>>>> I don't have such permission sadly. >>>>>> >>>>>> >>>>> Thank you anyway. >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kencochrane at gmail.com Sat Jun 8 04:08:12 2013 From: kencochrane at gmail.com (Ken Cochrane) Date: Fri, 7 Jun 2013 22:08:12 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: Message-ID: Ok, thanks Sent from my iPhone On Jun 7, 2013, at 9:56 PM, Brett Cannon wrote: > I might not get to it for a week, but I will get to it. > > > On Fri, Jun 7, 2013 at 5:26 PM, ken cochrane wrote: >> >>>> >>>>>>>> b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine >>>>>>>> >>>>>>>> b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. >>>>>>>> >>>>>>>> I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>>>>>>> >>>>>>>> I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. >>>>>>>> I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. >>>>>>>> >>>>>>>> I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. >>>>>>>> >>>>>>>> Two things: >>>>>>>> >>>>>>>> 1. Now that we have the pypi CDN up, do we still need this mirror? >>>>>>> >>>>>>> Honestly probably not. Mirrors are less important from a availability/speed side of things now and will likely move to being more useful for companies and such to use. >>>>>> >>>>>> OK, what would be the procedure for removing a mirror? Anyone know who is in charge of this mirror? I think Guido had it setup when he worked at Google, and google is paying for the costs of the mirror, but now that he doesn't work for Google, not sure who might be the contact person on that side. >>>>> >>>>> I've asked Guido who are admins on the account it to get it turned off. If he doesn't know I can try to find out internally. >>>> >>>> Brett, >>>> >>>> Here is the owner list that I can see on GAE >>>> >>>> - guido >>>> - kencochrane (me) >>>> - kumar.mcmillan >>>> - martin.v.loewis >>>> - r1chardj0n3s >>> >>> So Guido didn't even know he was an owner. =) >>> >>> In terms of shutting down the app, you will want to do two things. First is empty out the cron.yaml file; it should have nothing more than "cron:". After that you probably want to return 404 for everything; see http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. >>> >>> If you want, Ken, I can clone https://bitbucket.org/kencochrane/pypi-appengine and send you a pull request to do all of this since I recently did something similar for py3ksupport.appspot.com when I shut it down. >> >> Brett, >> >> If our goal is to shut it down, then yes please, if you can send a pull request that would be great.. We should also probably remove it from the pypi mirror pool before we do this so it no longer gets traffic sent it's way. >> >> >> >>>> >>>>>> >>>>>> If we can get it up to date again, I think it is fine, but an out of date mirror is not useful to anyone, and it could cause problems in the long run. >>>>>> >>>>>>>> >>>>>>>> 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. >>>>>>> >>>>>>> I don't have such permission sadly. >>>>>> >>>>>> Thank you anyway. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Sun Jun 9 00:57:57 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 9 Jun 2013 01:57:57 +0300 Subject: [Distutils] PyPI Download Counts In-Reply-To: <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> Message-ID: On Thu, May 30, 2013 at 6:56 PM, Donald Stufft wrote: > > On May 30, 2013, at 8:05 AM, Jim Fulton wrote: > > On Mon, May 27, 2013 at 3:27 AM, holger krekel wrote: > > Hi Donald, > > On Sun, May 26, 2013 at 20:08 -0400, Donald Stufft wrote: > > Hello! > > As you have have noticed the download counts on PyPI are no longer > updating. Originally this was due to an issue with the script that > processes these download counts. However I have now removed the download > counts from the PyPI webui and their use via the API is considered > deprecated. > > There are numerous reasons for their removal/deprecation some of which are: > - Technically hard to make work with the new CDN > - The CDN is being donated to the PSF, and the donated tier does > not offer any form of log access > > > What would be involved money/effort wise to get such access? > > > I didn't see an answer to this. > > Any idea how much this would cost? With access to logs, we could > compute download counts. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > > > Fastly has given us access to their streaming log support. > That's another deal. This is just what is expected in exchange for the "Real-time CDN by Fastly" phrase at the bottom of the page to make it reflect the reality. I'd say that not getting access to download stats from the start is a fail on the side of PSF or whoever involved in coordination with Fastly. There are many CDN providers out there and I suspect so far only Fastly was contacted. The primary responsibility of that coordinator is correctly sending the message for CDN provider that PyPI is a public exhibition of their service quality, and not a tax exemption for charity. I'd say Fastly should be interested to help with making us download stats exposed in a convenient API friendly way, because real-time stats is the key feature of their marketing advantage as I see it. Infrastructure will be setting up a secure method for receiving these logs > at which point PyPI can use them. > Is it done already? I am free to some degree to help with that. What is the current problem to tackle on? -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Sun Jun 9 14:53:36 2013 From: aclark at aclark.net (Alex Clark) Date: Sun, 9 Jun 2013 12:53:36 +0000 (UTC) Subject: [Distutils] PyPI Download Counts References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> Message-ID: anatoly techtonik gmail.com> writes: > > > I'd say that not getting access to download stats from the start is a fail on the side of PSF or whoever involved in coordination with Fastly. There are many CDN providers out there and I suspect so far only Fastly was contacted. The primary responsibility of that coordinator is correctly sending the message for CDN provider that PyPI is a public exhibition of their service quality, and not a tax exemption for charity. I'd say Fastly should be interested to help with making us download stats exposed in a convenient API friendly way, because real-time stats is the key feature of their marketing advantage as I see it. > > > Infrastructure will be setting up a secure method for receiving these logs at which point PyPI can use them. > > > > Is it done already? I am free to some degree to help with that. What is the current problem to tackle on? +1 But: this happens all the time in open source communities, even mature ones. Someone volunteers to perform some complex task and hand-waves-away some requirement they don't personally feel is important under the "too hard" or "I don't have time for that, I'm doing this" umbrella. Someone else comes along and says "wait, what happened to XXX?" at which point others join in to emphasize the importance of XXX. At this point, it's more clear whether or not XXX is actually important. But during this time: XXX is broken and many people on both sides are unhappy. GOLDEN RULE: That's why even in open source, the "golden rule of OPS" is still important: PLEASE DO NOT BREAK SHIT. Even if you are fixing other more important functionality and don't think XXX is important. Or if XXX and the thing you want to do cannot co-exist, then make sure you get extensive buy-in from a large percentage of folks before you consider removing XXX. If for no other reason, then you will most certainly draw unwanted attention to yourself (which is especially frustrating in the context of "trying to do good things"). Anyway, I've seen this so many times I don't even get angry anymore. Because I know everyone involved has nothing but the best intentions. In this case, I'd like to see the download count functionality restored. (I spent a full year building a website whose almost-sole-functionality relies on download counts existing and working properly.) So, please call me when they are back.:-) And if I can help in anyway to make that happen, please let me know. Alex > > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG python.org > http://mail.python.org/mailman/listinfo/distutils-sig > --- Alex Clark ? http://about.me/alex.clark From holger at merlinux.eu Sun Jun 9 15:20:35 2013 From: holger at merlinux.eu (holger krekel) Date: Sun, 9 Jun 2013 13:20:35 +0000 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> Message-ID: <20130609132035.GI7789@merlinux.eu> On Sun, Jun 09, 2013 at 01:57 +0300, anatoly techtonik wrote: > On Thu, May 30, 2013 at 6:56 PM, Donald Stufft wrote: > > > > > On May 30, 2013, at 8:05 AM, Jim Fulton wrote: > > > > On Mon, May 27, 2013 at 3:27 AM, holger krekel wrote: > > > > Hi Donald, > > > > On Sun, May 26, 2013 at 20:08 -0400, Donald Stufft wrote: > > > > Hello! > > > > As you have have noticed the download counts on PyPI are no longer > > updating. Originally this was due to an issue with the script that > > processes these download counts. However I have now removed the download > > counts from the PyPI webui and their use via the API is considered > > deprecated. > > > > There are numerous reasons for their removal/deprecation some of which are: > > - Technically hard to make work with the new CDN > > - The CDN is being donated to the PSF, and the donated tier does > > not offer any form of log access > > > > > > What would be involved money/effort wise to get such access? > > > > > > I didn't see an answer to this. > > > > Any idea how much this would cost? With access to logs, we could > > compute download counts. > > > > Jim > > > > -- > > Jim Fulton > > http://www.linkedin.com/in/jimfulton > > > > > > Fastly has given us access to their streaming log support. > > > > That's another deal. This is just what is expected in exchange for the > "Real-time CDN by Fastly" phrase at the bottom of the page to make it > reflect the reality. > > I'd say that not getting access to download stats from the start is a fail > on the side of PSF or whoever involved in coordination with Fastly. I also was surprised when download counts were gone but download stats aren't very reliable indicators, anyway. To begin with, one might configure a CI system to cache pypi packages, another might re-download all the time. Some popular packages are rather used from system distros, others downloaded from pypi. And there are many more issues. Seems like there are not overwhelmingly many people unhappy about its removal. Doesn't mean they shouldn't be reinstanted, just saying. > There > are many CDN providers out there and I suspect so far only Fastly was > contacted. The primary responsibility of that coordinator is correctly > sending the message for CDN provider that PyPI is a public exhibition of > their service quality, and not a tax exemption for charity. I'd say Fastly > should be interested to help with making us download stats exposed in a > convenient API friendly way, because real-time stats is the key feature of > their marketing advantage as I see it. FYI Fastly has been responding swiftly and responsibly so far, as far as i could follow it. And Donald has been helping people on various fronts (not only Fastly related) and been the main driver of the CDN move, thankfully. Things are starting to work more smoothly now and i guess some bumps in the road were unavoidable because they only show up in real life. cheers, holger > Infrastructure will be setting up a secure method for receiving these logs > > at which point PyPI can use them. > > > > Is it done already? I am free to some degree to help with that. What is the > current problem to tackle on? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From jaraco at jaraco.com Sun Jun 9 18:22:32 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 9 Jun 2013 16:22:32 +0000 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI Message-ID: On behalf of the PYPA, I'm excited to announce that Setuptools 0.7 is now official and complete. Released to PyPI, Setuptools 0.7.2 is now available for all to see by default (https://pypi.python.org/pypi/setuptools/). Users of Setuptools 0.6 may upgrade by simply running `easy_install -U setuptools`. Additionally, I released Distribute 0.7 to PyPI. This means that users who have Distribute 0.6.x can upgrade to setuptools 0.7 by simply invoking `easy_install -U distribute` (or `easy_install "distribute>=0.7"`). Windows users should use `easy_install-script -U distribute` (to avoid "file in use" errors on easy_install.exe). The documentation for Setuptools 0.7.2 has been uploaded to https://pythonhosted.org/setuptools and includes the notes about the merge in addition to the official documentation. Please report any issues at the project page (https://bitbucket.org/pypa/setuptools). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From ncoghlan at gmail.com Sun Jun 9 18:41:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Jun 2013 02:41:28 +1000 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: Yay, great news! /me adds updating the packaging essay (both copies) to his TODO list :) Cheers, Nick. From tseaver at palladion.com Sun Jun 9 19:13:34 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 09 Jun 2013 13:13:34 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/2013 12:22 PM, Jason R. Coombs wrote: > On behalf of the PYPA, I'm excited to announce that Setuptools 0.7 is > now official and complete. > > Released to PyPI, Setuptools 0.7.2 is now available for all to see by > default (https://pypi.python.org/pypi/setuptools/). Users of > Setuptools 0.6 may upgrade by simply running `easy_install -U > setuptools`. > > Additionally, I released Distribute 0.7 to PyPI. This means that users > who have Distribute 0.6.x can upgrade to setuptools 0.7 by simply > invoking `easy_install -U distribute` (or `easy_install > "distribute>=0.7"`). Windows users should use `easy_install-script -U > distribute` (to avoid "file in use" errors on easy_install.exe). > > The documentation for Setuptools 0.7.2 has been uploaded to > https://pythonhosted.org/setuptools and includes the notes about the > merge in addition to the official documentation. > > Please report any issues at the project page > (https://bitbucket.org/pypa/setuptools). Thanks for all the work that went into this release. I'm working to get buildout 2.0 to use the new setuptools, and need to encode a more-or-less permalink URL for 'ez_setup.py' -- do we not have a better URL for it than the Bitbucket 'raw' source? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG0t74ACgkQ+gerLs4ltQ6Q5wCg2hMoLBxfv6NIR/DTOUjdBNei kVUAnjZ9ua2gRpwXyeRw/i1CIBPeV9e0 =Ksc7 -----END PGP SIGNATURE----- From liamk at numenet.com Sun Jun 9 19:23:47 2013 From: liamk at numenet.com (Liam Kirsher) Date: Sun, 09 Jun 2013 10:23:47 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 Message-ID: <51B4BA23.2060009@numenet.com> Hi -- I'm hoping for some help with an error I'm suddenly getting with distribute, setuptools. It had been working fine, and seemingly overnight it stopped working. I really don't think I changed anything. This is part of a chef recipe, which also installs virtualenv. It appears that a new version 0.7.2 is downloaded, but it fails to install, and then the rollback fails, as well. Not sure quite what is wrong here and how to fix it! Liam > Using version 0.7.2 (newest of versions: 0.7.2, 0.7.2, 0.7.1, 0.7) > Downloading from URL > https://pypi.python.org/packages/source/s/setuptools/setuptools-0.7.2.tar.gz#md5=de44cd90f8a1c713d6c2bff67bbca65d > (from https://pypi.python.org/simple/setuptools/) > Running setup.py egg_info for package setuptools > > running egg_info > creating pip-egg-info/setuptools.egg-info > writing requirements to pip-egg-info/setuptools.egg-info/requires.txt > writing pip-egg-info/setuptools.egg-info/PKG-INFO > writing top-level names to > pip-egg-info/setuptools.egg-info/top_level.txt > writing dependency_links to > pip-egg-info/setuptools.egg-info/dependency_links.txt > writing entry points to > pip-egg-info/setuptools.egg-info/entry_points.txt > writing requirements to pip-egg-info/setuptools.egg-info/requires.txt > writing pip-egg-info/setuptools.egg-info/PKG-INFO > writing top-level names to > pip-egg-info/setuptools.egg-info/top_level.txt > writing dependency_links to > pip-egg-info/setuptools.egg-info/dependency_links.txt > writing entry points to > pip-egg-info/setuptools.egg-info/entry_points.txt > writing manifest file 'pip-egg-info/setuptools.egg-info/SOURCES.txt' > warning: manifest_maker: standard file '-c' not found > > > reading manifest file 'pip-egg-info/setuptools.egg-info/SOURCES.txt' > reading manifest template 'MANIFEST.in' > writing manifest file 'pip-egg-info/setuptools.egg-info/SOURCES.txt' > Source in /tmp/pip-build-root/setuptools has version 0.7.2, which > satisfies requirement setuptools>=0.7 (from distribute->supervisor) > skipping extra ssl:sys_platform=='win32' > skipping extra ssl:sys_platform=='win32' and python_version=='2.4' > skipping extra certs > skipping extra ssl:python_version in '2.4, 2.5' > Installing collected packages: distribute, setuptools > > Found existing installation: distribute 0.6.45 > > Uninstalling distribute: > > Removing file or directory > /usr/local/lib/python2.7/dist-packages/distribute-0.6.45-py2.7.egg > Removing pth entries from > /usr/local/lib/python2.7/dist-packages/easy-install.pth: > Removing entry: ./distribute-0.6.45-py2.7.egg > Successfully uninstalled distribute > > Running setup.py install for distribute > > Running command /usr/bin/python -c "import > setuptools;__file__='/tmp/pip-build-root/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-5tzG3v-record/install-record.txt > --single-version-externally-managed > Traceback (most recent call last): > > File "", line 1, in > > ImportError: No module named setuptools > > Complete output from command /usr/bin/python -c "import > setuptools;__file__='/tmp/pip-build-root/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-5tzG3v-record/install-record.txt > --single-version-externally-managed: > > Traceback (most recent call last): > > File "", line 1, in > > ImportError: No module named setuptools > > ---------------------------------------- > > Rolling back uninstall of distribute > > Replacing > /usr/local/lib/python2.7/dist-packages/distribute-0.6.45-py2.7.egg > Exception: > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py", > line 139, in main > status = self.run(options, args) > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py", > line 271, in run > requirement_set.install(install_options, global_options, > root=options.root_path) > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", > line 1189, in install > requirement.rollback_uninstall() > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", > line 500, in rollback_uninstall > self.uninstalled.rollback() > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", > line 1537, in rollback > pth.rollback() > AttributeError: 'str' object has no attribute 'rollback' -- Liam Kirsher PGP: http://liam.numenet.com/pgp/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaraco at jaraco.com Sun Jun 9 19:49:18 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 9 Jun 2013 17:49:18 +0000 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: Is the bitbucket 'raw' source not suitable? It uses the version tag to indicate precisely which file is appropriate, which means we don't have to redundantly host that file elsewhere. Oh, I think I see - there's no link that doesn't change - you want a permalink for the latest stable release. I hadn't considered that use case, but you're right - we do want that. Perhaps the file could be added to pythonhosted.org alongside the documentation. Or maybe it belongs on bitbucket in `downloads`. I need to figure out which is more straightforward w.r.t. authentication. The distribute process was messy because it involved SSH and keys that had to be manually managed. I'm leaning toward uploading it to BitBucket downloads as part of the release script. I'll put ez_setup.py for 0.7.2 in downloads for now. You can assume that's where the permalink will be. If anyone has a better suggestion, please raise it. > -----Original Message----- > From: Distutils-SIG [mailto:distutils-sig- > bounces+jaraco=jaraco.com at python.org] On Behalf Of Tres Seaver > Sent: Sunday, 09 June, 2013 13:14 > To: Distutils-Sig at Python.Org > Subject: Re: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility > wrapper) on PyPI > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/09/2013 12:22 PM, Jason R. Coombs wrote: > > On behalf of the PYPA, I'm excited to announce that Setuptools 0.7 is > > now official and complete. > > > > Released to PyPI, Setuptools 0.7.2 is now available for all to see by > > default (https://pypi.python.org/pypi/setuptools/). Users of > > Setuptools 0.6 may upgrade by simply running `easy_install -U > > setuptools`. > > > > Additionally, I released Distribute 0.7 to PyPI. This means that users > > who have Distribute 0.6.x can upgrade to setuptools 0.7 by simply > > invoking `easy_install -U distribute` (or `easy_install > > "distribute>=0.7"`). Windows users should use `easy_install-script -U > > distribute` (to avoid "file in use" errors on easy_install.exe). > > > > The documentation for Setuptools 0.7.2 has been uploaded to > > https://pythonhosted.org/setuptools and includes the notes about the > > merge in addition to the official documentation. > > > > Please report any issues at the project page > > (https://bitbucket.org/pypa/setuptools). > > > Thanks for all the work that went into this release. I'm working to get > buildout 2.0 to use the new setuptools, and need to encode a more-or-less > permalink URL for 'ez_setup.py' -- do we not have a better URL for it than > the Bitbucket 'raw' source? > > > > Tres. > - -- > ========================================================== > ========= > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlG0t74ACgkQ+gerLs4ltQ6Q5wCg2hMoLBxfv6NIR/DTOUjdBNe > i > kVUAnjZ9ua2gRpwXyeRw/i1CIBPeV9e0 > =Ksc7 > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From jaraco at jaraco.com Sun Jun 9 20:07:23 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 9 Jun 2013 18:07:23 +0000 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: <51B4BA23.2060009@numenet.com> References: <51B4BA23.2060009@numenet.com> Message-ID: Hi Liam, Sorry for the trouble. The cause is rooted in the latest updates to Setuptools and Distribute on PyPI which were launched today. I believe what's happening here is pip is installing Distribute 0.7, which triggers the uninstallation of Distribute, but Distribute 0.7 (a compatibility wrapper) depends on a setuptools module to be in place to install. I'm unsure about the failed rollback. That looks like a separate issue in pip. Based on your report, I'm going to remove the Distribute-0.7 wrapper from PyPI until this issue is resolved. That is done. In order to repair those envs, you may simply be able to re-run the chef code over the environments and they may repair naturally. If they don't, you may need to re-initialize the virtualenv (run virtualenv again on that path) in order to revive distribute in that environment. I hope the pip or virtualenv maintainers can comment as well. Regards, Jason From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Liam Kirsher Sent: Sunday, 09 June, 2013 13:24 To: distutils-sig at python.org Subject: [Distutils] Setuptools/Distribute error with 0.7.2 Hi -- I'm hoping for some help with an error I'm suddenly getting with distribute, setuptools. It had been working fine, and seemingly overnight it stopped working. I really don't think I changed anything. This is part of a chef recipe, which also installs virtualenv. It appears that a new version 0.7.2 is downloaded, but it fails to install, and then the rollback fails, as well. Not sure quite what is wrong here and how to fix it! Liam Using version 0.7.2 (newest of versions: 0.7.2, 0.7.2, 0.7.1, 0.7) Downloading from URL https://pypi.python.org/packages/source/s/setuptools/setuptools-0.7.2.tar.gz #md5=de44cd90f8a1c713d6c2bff67bbca65d (from https://pypi.python.org/simple/setuptools/) Running setup.py egg_info for package setuptools running egg_info creating pip-egg-info/setuptools.egg-info writing requirements to pip-egg-info/setuptools.egg-info/requires.txt writing pip-egg-info/setuptools.egg-info/PKG-INFO writing top-level names to pip-egg-info/setuptools.egg-info/top_level.txt writing dependency_links to pip-egg-info/setuptools.egg-info/dependency_links.txt writing entry points to pip-egg-info/setuptools.egg-info/entry_points.txt writing requirements to pip-egg-info/setuptools.egg-info/requires.txt writing pip-egg-info/setuptools.egg-info/PKG-INFO writing top-level names to pip-egg-info/setuptools.egg-info/top_level.txt writing dependency_links to pip-egg-info/setuptools.egg-info/dependency_links.txt writing entry points to pip-egg-info/setuptools.egg-info/entry_points.txt writing manifest file 'pip-egg-info/setuptools.egg-info/SOURCES.txt' warning: manifest_maker: standard file '-c' not found reading manifest file 'pip-egg-info/setuptools.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pip-egg-info/setuptools.egg-info/SOURCES.txt' Source in /tmp/pip-build-root/setuptools has version 0.7.2, which satisfies requirement setuptools>=0.7 (from distribute->supervisor) skipping extra ssl:sys_platform=='win32' skipping extra ssl:sys_platform=='win32' and python_version=='2.4' skipping extra certs skipping extra ssl:python_version in '2.4, 2.5' Installing collected packages: distribute, setuptools Found existing installation: distribute 0.6.45 Uninstalling distribute: Removing file or directory /usr/local/lib/python2.7/dist-packages/distribute-0.6.45-py2.7.egg Removing pth entries from /usr/local/lib/python2.7/dist-packages/easy-install.pth: Removing entry: ./distribute-0.6.45-py2.7.egg Successfully uninstalled distribute Running setup.py install for distribute Running command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build-root/distribute/setup.py';exec(compile(o pen(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-5tzG3v-record/install-record.txt --single-version-externally-managed Traceback (most recent call last): File "", line 1, in ImportError: No module named setuptools Complete output from command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build-root/distribute/setup.py';exec(compile(o pen(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-5tzG3v-record/install-record.txt --single-version-externally-managed: Traceback (most recent call last): File "", line 1, in ImportError: No module named setuptools ---------------------------------------- Rolling back uninstall of distribute Replacing /usr/local/lib/python2.7/dist-packages/distribute-0.6.45-py2.7.egg Exception: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/basecommand. py", line 139, in main status = self.run(options, args) File "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/commands/ins tall.py", line 271, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 1189, in install requirement.rollback_uninstall() File "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 500, in rollback_uninstall self.uninstalled.rollback() File "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 1537, in rollback pth.rollback() AttributeError: 'str' object has no attribute 'rollback' -- Liam Kirsher PGP: http://liam.numenet.com/pgp/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From donald at stufft.io Sun Jun 9 20:11:42 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 9 Jun 2013 14:11:42 -0400 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <51B4BA23.2060009@numenet.com> Message-ID: On Jun 9, 2013, at 2:07 PM, "Jason R. Coombs" wrote: > I believe what?s happening here is pip is installing Distribute 0.7, which triggers the uninstallation of Distribute, but Distribute 0.7 (a compatibility wrapper) depends on a setuptools module to be in place to install. If I recall pip hasn't handled the upgrade of distribute very well for awhile now but I can't see to find an existing ticket for it. There is a new one an hour ago but I assume it stems from this issue. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From julian at grayvines.com Sun Jun 9 20:20:57 2013 From: julian at grayvines.com (Julian Berman) Date: Sun, 9 Jun 2013 14:20:57 -0400 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 Message-ID: (I was also getting this until I used easy_install to upgrade) What is supposed to happen with stuff that used to do `use_setuptools()` in their setup.py now? I upgraded to setuptools 0.7.2, but now if I try to install a thing that has that in the setup.py, it seems to want to try and install distribute, and I get: Downloading/unpacking pudb (from -r /Users/Julian/.local/share/virtualenvs/requirements-every-virtualenv.txt (line 2)) Using download cache from /Users/Julian/Library/Caches/pip/https%3A%2F% 2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpudb%2Fpudb-2013.2.tar.gz Running setup.py egg_info for package pudb Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3 Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 Building a Distribute egg in /Users/Julian/.local/share/virtualenvs/great/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg Traceback (most recent call last): File "", line 16, in File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. Complete output from command python setup.py egg_info: Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3 Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 Building a Distribute egg in /Users/Julian/.local/share/virtualenvs/great/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg Traceback (most recent call last): File "", line 16, in File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. ---------------------------------------- Command python setup.py egg_info failed with error code 1 in /Users/Julian/.local/share/virtualenvs/great/build/pudb S -------------- next part -------------- An HTML attachment was scrubbed... URL: From liamk at numenet.com Sun Jun 9 20:29:35 2013 From: liamk at numenet.com (Liam Kirsher) Date: Sun, 09 Jun 2013 11:29:35 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <51B4BA23.2060009@numenet.com> Message-ID: <51B4C98F.30209@numenet.com> Thanks! That's a relief. I re-ran it and it worked. On 06/09/2013 11:07 AM, Jason R. Coombs wrote: > > Hi Liam, > > Sorry for the trouble. The cause is rooted in the > latest updates to Setuptools and Distribute on PyPI which were > launched today. > > > > I believe what's happening here is pip is installing > Distribute 0.7, which triggers the uninstallation of Distribute, but > Distribute 0.7 (a compatibility wrapper) depends on a setuptools > module to be in place to install. > > > > I'm unsure about the failed rollback. That looks like > a separate issue in pip. > > > > Based on your report, I'm going to remove the > Distribute-0.7 wrapper from PyPI until this issue is resolved. That is > done. > > > > In order to repair those envs, you may simply be able > to re-run the chef code over the environments and they may repair > naturally. If they don't, you may need to re-initialize the virtualenv > (run virtualenv again on that path) in order to revive distribute in > that environment. > > > > I hope the pip or virtualenv maintainers can comment > as well. > > > > Regards, > > Jason > > > > *From:*Distutils-SIG > [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] *On Behalf > Of *Liam Kirsher > *Sent:* Sunday, 09 June, 2013 13:24 > *To:* distutils-sig at python.org > *Subject:* [Distutils] Setuptools/Distribute error with 0.7.2 > > > > Hi -- > I'm hoping for some help with an error I'm suddenly getting with > distribute, setuptools. > It had been working fine, and seemingly overnight it stopped working. > I really don't think I changed anything. > This is part of a chef recipe, which also installs virtualenv. > It appears that a new version 0.7.2 is downloaded, but it fails to > install, and then the rollback fails, as well. > > Not sure quite what is wrong here and how to fix it! > > Liam > > > Using version 0.7.2 (newest of versions: 0.7.2, 0.7.2, 0.7.1, 0.7) > Downloading from URL > https://pypi.python.org/packages/source/s/setuptools/setuptools-0.7.2.tar.gz#md5=de44cd90f8a1c713d6c2bff67bbca65d > (from https://pypi.python.org/simple/setuptools/) > Running setup.py egg_info for package setuptools > > running egg_info > creating pip-egg-info/setuptools.egg-info > writing requirements to > pip-egg-info/setuptools.egg-info/requires.txt > writing pip-egg-info/setuptools.egg-info/PKG-INFO > writing top-level names to > pip-egg-info/setuptools.egg-info/top_level.txt > writing dependency_links to > pip-egg-info/setuptools.egg-info/dependency_links.txt > writing entry points to > pip-egg-info/setuptools.egg-info/entry_points.txt > writing requirements to > pip-egg-info/setuptools.egg-info/requires.txt > writing pip-egg-info/setuptools.egg-info/PKG-INFO > writing top-level names to > pip-egg-info/setuptools.egg-info/top_level.txt > writing dependency_links to > pip-egg-info/setuptools.egg-info/dependency_links.txt > writing entry points to > pip-egg-info/setuptools.egg-info/entry_points.txt > writing manifest file > 'pip-egg-info/setuptools.egg-info/SOURCES.txt' > warning: manifest_maker: standard file '-c' not found > > > reading manifest file > 'pip-egg-info/setuptools.egg-info/SOURCES.txt' > reading manifest template 'MANIFEST.in' > writing manifest file > 'pip-egg-info/setuptools.egg-info/SOURCES.txt' > Source in /tmp/pip-build-root/setuptools has version 0.7.2, > which satisfies requirement setuptools>=0.7 (from > distribute->supervisor) > skipping extra ssl:sys_platform=='win32' > skipping extra ssl:sys_platform=='win32' and python_version=='2.4' > skipping extra certs > skipping extra ssl:python_version in '2.4, 2.5' > Installing collected packages: distribute, setuptools > > Found existing installation: distribute 0.6.45 > > Uninstalling distribute: > > Removing file or directory > /usr/local/lib/python2.7/dist-packages/distribute-0.6.45-py2.7.egg > Removing pth entries from > /usr/local/lib/python2.7/dist-packages/easy-install.pth: > Removing entry: ./distribute-0.6.45-py2.7.egg > Successfully uninstalled distribute > > Running setup.py install for distribute > > Running command /usr/bin/python -c "import > setuptools;__file__='/tmp/pip-build-root/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-5tzG3v-record/install-record.txt > --single-version-externally-managed > Traceback (most recent call last): > > File "", line 1, in > > ImportError: No module named setuptools > > Complete output from command /usr/bin/python -c "import > setuptools;__file__='/tmp/pip-build-root/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', > '\n'), __file__, 'exec'))" install --record > /tmp/pip-5tzG3v-record/install-record.txt > --single-version-externally-managed: > > Traceback (most recent call last): > > File "", line 1, in > > ImportError: No module named setuptools > > ---------------------------------------- > > Rolling back uninstall of distribute > > Replacing > /usr/local/lib/python2.7/dist-packages/distribute-0.6.45-py2.7.egg > Exception: > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py", > line 139, in main > status = self.run(options, args) > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py", > line 271, in run > requirement_set.install(install_options, global_options, > root=options.root_path) > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", > line 1189, in install > requirement.rollback_uninstall() > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", > line 500, in rollback_uninstall > self.uninstalled.rollback() > File > "/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg/pip/req.py", > line 1537, in rollback > pth.rollback() > AttributeError: 'str' object has no attribute 'rollback' > > > > -- > Liam Kirsher > PGP: http://liam.numenet.com/pgp/ -- Liam Kirsher PGP: http://liam.numenet.com/pgp/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jun 9 20:33:17 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 9 Jun 2013 14:33:17 -0400 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130609132035.GI7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <20130609132035.GI7789@merlinux.eu> Message-ID: On Jun 9, 2013, at 9:20 AM, holger krekel wrote: >> There >> are many CDN providers out there and I suspect so far only Fastly was >> contacted. The primary responsibility of that coordinator is correctly >> sending the message for CDN provider that PyPI is a public exhibition of >> their service quality, and not a tax exemption for charity. I'd say Fastly >> should be interested to help with making us download stats exposed in a >> convenient API friendly way, because real-time stats is the key feature of >> their marketing advantage as I see it. > > FYI Fastly has been responding swiftly and responsibly so far, as far as > i could follow it. And Donald has been helping people on various fronts > (not only Fastly related) and been the main driver of the CDN move, > thankfully. Things are starting to work more smoothly now and i guess > some bumps in the road were unavoidable because they only show up in > real life. Fastly has been a dream to work with. They've been fast at fixing and diagnosing issues, have helped tune the config to get a higher hit rate, and when they heard that people were upset that download counts had to be turned off they offered the logging support to be turned on for our account. The infrastructure is setup to receive the logs (and intact was receiving logs for a day or so) but upgrades to the VM that runs PyPI needs to occur before we can continue receiving them. The disk drive that PyPI has is to small to handle the volume of request data that is coming in from Fastly and it quickly filled up in under 24 hours. Upgrading that space requires powering off the VM so we (The Infra team) are working on doing that, ideally without downtime on PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Sun Jun 9 20:38:14 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 9 Jun 2013 20:38:14 +0200 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: I'm dancing a small victory-dance in my chair. On Sun, Jun 9, 2013 at 6:22 PM, Jason R. Coombs wrote: > On behalf of the PYPA, I?m excited to announce that Setuptools 0.7 is now > official and complete. > > > > Released to PyPI, Setuptools 0.7.2 is now available for all to see by > default (https://pypi.python.org/pypi/setuptools/). Users of Setuptools 0.6 > may upgrade by simply running `easy_install -U setuptools`. > > > > Additionally, I released Distribute 0.7 to PyPI. This means that users who > have Distribute 0.6.x can upgrade to setuptools 0.7 by simply invoking > `easy_install -U distribute` (or `easy_install "distribute>=0.7?`). Windows > users should use `easy_install-script -U distribute` (to avoid ?file in use? > errors on easy_install.exe). > > > > The documentation for Setuptools 0.7.2 has been uploaded to > https://pythonhosted.org/setuptools and includes the notes about the merge > in addition to the official documentation. > > > > Please report any issues at the project page > (https://bitbucket.org/pypa/setuptools). > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From donald at stufft.io Sun Jun 9 20:41:35 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 9 Jun 2013 14:41:35 -0400 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> Message-ID: <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> On Jun 9, 2013, at 8:53 AM, Alex Clark wrote: > > But: this happens all the time in open source communities, even > mature ones. Someone volunteers to perform some complex task and > hand-waves-away some requirement they don't personally feel is important > under the "too hard" or "I don't have time for that, I'm doing this" umbrella. > > Someone else comes along and says "wait, what happened to XXX?" at which > point others join in to emphasize the importance of XXX. At this point, it's > more clear whether or not XXX is actually important. But during this time: > XXX is broken and many people on both sides are unhappy. > > GOLDEN RULE: That's why even in open source, the "golden rule of OPS" is > still important: PLEASE DO NOT BREAK SHIT. Even if you are fixing other more > important functionality and don't think XXX is important. Or if XXX and the > thing you want to do cannot co-exist, then make sure you get extensive > buy-in from a large percentage of folks before you consider removing XXX. If > for no other reason, then you will most certainly draw unwanted attention to > yourself (which is especially frustrating in the context of "trying to do > good things"). > > > Anyway, I've seen this so many times I don't even get angry anymore. Because > I know everyone involved has nothing but the best intentions. In this case, > I'd like to see the download count functionality restored. (I spent a full > year building a website whose almost-sole-functionality relies on download > counts existing and working properly.) So, please call me when they are > back.:-) And if I can help in anyway to make that happen, please let me know. > PyPI's central purpose is to act as a repository and index of packages. Download counts are an auxiliary feature. Prior to the CDN there were large swathes of people, primarily in non US locations, who simply could not use PyPI because of bad routes between them and PyPI. Some folks were getting download speeds more reminiscent of dialup instead of Broadband. Moving to a CDN made PyPI reasonably function again for those people. So yes. I broke Download counts because they were not more important than people being able to actually use PyPI to install from. And, thanks to Fastly generously giving us a access to streaming logs, will be bringing them back once the issues are sorted out. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Sun Jun 9 20:43:06 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 9 Jun 2013 11:43:06 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <51B4BA23.2060009@numenet.com> Message-ID: > If I recall pip hasn't handled the upgrade of distribute very well for > awhile now but I can't see to find an existing ticket for it. > the older issue is that pip can't upgrade distribute in python 3 (related to distribute requiring 2to3) https://github.com/pypa/pip/issues/650 btw, pip-1.4 (unreleased) has new logic to skip even trying, because the problem prevents upgrades and wheel building for projects like "pyramid" (which formally depend on setuptools) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaraco at jaraco.com Sun Jun 9 20:29:01 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sun, 9 Jun 2013 18:29:01 +0000 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: Message-ID: <05c820d2d5984850b6dd8543ae3af5f9@BLUPR06MB003.namprd06.prod.outlook.com> Hi Julian, I believe for those packages that call `use_setuptools()` from `distribute_setup.py`, the only option to support them is to install Distribute 0.7 (the stub package), which I believe will satisfy the use_setuptools() call so it doesn't attempt to install Distribute 0.6. As mentioned in another post, I've temporarily pulled Distribute 0.7 from PyPI until we can determine a way to avoid the pip upgrade issue. In the meantime, if you install Distribute 0.7 from https://bitbucket.org/pypa/setuptools/downloads, does that solve your issue? Regards, Jason From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Julian Berman Sent: Sunday, 09 June, 2013 14:21 To: distutils-sig at python.org Subject: Re: [Distutils] Setuptools/Distribute error with 0.7.2 (I was also getting this until I used easy_install to upgrade) What is supposed to happen with stuff that used to do `use_setuptools()` in their setup.py now? I upgraded to setuptools 0.7.2, but now if I try to install a thing that has that in the setup.py, it seems to want to try and install distribute, and I get: Downloading/unpacking pudb (from -r /Users/Julian/.local/share/virtualenvs/requirements-every-virtualenv.txt (line 2)) Using download cache from /Users/Julian/Library/Caches/pip/https%3A%2F%2Fpypi.python.org %2Fpackages%2Fsource%2Fp%2Fpudb%2Fpudb-2013.2.tar.gz Running setup.py egg_info for package pudb Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3 Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 Building a Distribute egg in /Users/Julian/.local/share/virtualenvs/great/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/set uptools-0.7.2-py2.7.egg /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py 2.7.egg Traceback (most recent call last): File "", line 16, in File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. Complete output from command python setup.py egg_info: Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3 Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 Building a Distribute egg in /Users/Julian/.local/share/virtualenvs/great/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribu te-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/set uptools-0.7.2-py2.7.egg /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py 2.7.egg Traceback (most recent call last): File "", line 16, in File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. ---------------------------------------- Command python setup.py egg_info failed with error code 1 in /Users/Julian/.local/share/virtualenvs/great/build/pudb S -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From qwcode at gmail.com Sun Jun 9 21:47:10 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 9 Jun 2013 12:47:10 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <51B4BA23.2060009@numenet.com> Message-ID: > Hi Liam,**** > > Sorry for the trouble. The cause is rooted in the latest > updates to Setuptools and Distribute on PyPI which were launched today.*** > * > > ** ** > > I believe what?s happening here is pip is installing > Distribute 0.7, which triggers the uninstallation of Distribute, but > Distribute 0.7 (a compatibility wrapper) depends on a setuptools module to > be in place to install.**** > > ** ** > > I?m unsure about the failed rollback. That looks like a > separate issue in pip. > the simplest answer IMO is to not use pip to upgrade distribute/setuptools right now. It was only working in python2 before now anyway. ( https://github.com/pypa/pip/issues/650) pip is fundamentally dependent on setuptools to perform upgrades. the best fix I think is to move faster on making pip "PEP439 compliant" - i.e. have pip be able to at least install from wheels w/o needing setuptools, which would remove the bootstrap headache - see https://github.com/pypa/pip/issues/863 - this could likely involve pip replacing it's use of pkg_resources with distlib (https://github.com/pypa/pip/pull/909) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Sun Jun 9 21:46:30 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 9 Jun 2013 12:46:30 -0700 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: <39538F60-678B-42CB-B17D-B62B58409C07@stufft.io> Message-ID: <5E6CBF13-A2B9-4699-A2B9-3C046F489C25@coderanger.net> On Jun 7, 2013, at 2:34 PM, Noah Kantrowitz wrote: > > On Jun 7, 2013, at 2:27 PM, Donald Stufft wrote: > >> >> On Jun 7, 2013, at 5:26 PM, ken cochrane wrote: >> >>> >>> >>>> b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine >>>> >>>> b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. >>>> >>>> I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>>> >>>> I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. >>>> I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. >>>> >>>> I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. >>>> >>>> Two things: >>>> >>>> 1. Now that we have the pypi CDN up, do we still need this mirror? >>> >>> Honestly probably not. Mirrors are less important from a availability/speed side of things now and will likely move to being more useful for companies and such to use. >>> >>> OK, what would be the procedure for removing a mirror? Anyone know who is in charge of this mirror? I think Guido had it setup when he worked at Google, and google is paying for the costs of the mirror, but now that he doesn't work for Google, not sure who might be the contact person on that side. >>> >>> I've asked Guido who are admins on the account it to get it turned off. If he doesn't know I can try to find out internally. >>> >>> >>> Brett, >>> >>> Here is the owner list that I can see on GAE >>> >>> - guido >>> - kencochrane (me) >>> - kumar.mcmillan >>> - martin.v.loewis >>> - r1chardj0n3s >>> >>> So Guido didn't even know he was an owner. =) >>> >>> In terms of shutting down the app, you will want to do two things. First is empty out the cron.yaml file; it should have nothing more than "cron:". After that you probably want to return 404 for everything; see http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. >>> >>> If you want, Ken, I can clone https://bitbucket.org/kencochrane/pypi-appengine and send you a pull request to do all of this since I recently did something similar for py3ksupport.appspot.com when I shut it down. >>> >>> Brett, >>> >>> If our goal is to shut it down, then yes please, if you can send a pull request that would be great.. We should also probably remove it from the pypi mirror pool before we do this so it no longer gets traffic sent it's way. >>> >>> >>> >>> >>> >>> If we can get it up to date again, I think it is fine, but an out of date mirror is not useful to anyone, and it could cause problems in the long run. >>> >>>> >>>> 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. >>> >>> I don't have such permission sadly. >>> >>> >>> Thank you anyway. >>> >>> >>> >>> >>> >> >> Paging Noah to kill the DNS Unless there's any objections to removing it? > > If no one complains in the next 24-ish hours I'll just point it back at pypi.python.org like we did with d.pypi. This is now complete. In another 24 hours there should be no traffic to the GAE app and it can just be archived or deleted. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: Message signed with OpenPGP using GPGMail URL: From aclark at aclark.net Sun Jun 9 22:04:45 2013 From: aclark at aclark.net (Alex Clark) Date: Sun, 9 Jun 2013 20:04:45 +0000 (UTC) Subject: [Distutils] PyPI Download Counts References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > So yes. I broke Download counts because they were not more important than > people being able to actually use PyPI to install from. FWIW: You missed the moral of the story: when you make a decision like this, someone will *always* disagree with you (even over the most trivial things). And even if they don't, they may disagree with your approach (e.g. why not sort problems with download counts before enabling the CDN) So the only way to make everyone happy is to consider everyone who will be affected by your actions, before you take action. (I'm not that concerned with this particular incident, just fascinated by open source community relations in general.) Alex > > ----------------- > Donald Stufft > PGP:?0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From noah at coderanger.net Sun Jun 9 22:08:15 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 9 Jun 2013 13:08:15 -0700 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> Message-ID: On Jun 9, 2013, at 1:04 PM, Alex Clark wrote: > Donald Stufft stufft.io> writes: > >> So yes. I broke Download counts because they were not more important than >> people being able to actually use PyPI to install from. > > > FWIW: > > > You missed the moral of the story: when you make a decision like this, > someone will *always* disagree with you (even over the most trivial things). > And even if they don't, they may disagree with your approach (e.g. why not > sort problems with download counts before enabling the CDN) So the only way > to make everyone happy is to consider everyone who will be affected by your > actions, before you take action. There is another way, make awesome and wait for history to determine who was happy :) --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: Message signed with OpenPGP using GPGMail URL: From regebro at gmail.com Sun Jun 9 22:09:25 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 9 Jun 2013 22:09:25 +0200 Subject: [Distutils] PyPI Download Counts In-Reply-To: <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> Message-ID: On Sun, Jun 9, 2013 at 8:41 PM, Donald Stufft wrote: > So yes. I broke Download counts because they were not more important than > people being able to actually use PyPI to install from. I approve of this message. //Lennart From kencochrane at gmail.com Sun Jun 9 22:11:18 2013 From: kencochrane at gmail.com (Ken Cochrane) Date: Sun, 9 Jun 2013 16:11:18 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: <5E6CBF13-A2B9-4699-A2B9-3C046F489C25@coderanger.net> References: <39538F60-678B-42CB-B17D-B62B58409C07@stufft.io> <5E6CBF13-A2B9-4699-A2B9-3C046F489C25@coderanger.net> Message-ID: On Jun 9, 2013, at 3:46 PM, Noah Kantrowitz wrote: > > On Jun 7, 2013, at 2:34 PM, Noah Kantrowitz wrote: > >> >> On Jun 7, 2013, at 2:27 PM, Donald Stufft wrote: >> >>> >>> On Jun 7, 2013, at 5:26 PM, ken cochrane wrote: >>> >>>> >>>> >>>>> b.pypi.python.org is an official mirror that runs on Google App engine, and it uses a special mirror package built just for GAE. Code for it is found here. https://bitbucket.org/loewis/pypi-appengine >>>>> >>>>> b.pypi.python.org has been broken for over 104 days according to http://www.pypi-mirrors.org, and this is because of an issue when we switched pypi over to serving over SSL. >>>>> >>>>> I have submitted a pull request to fix this. https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >>>>> >>>>> I am one of the maintainers of b.pypi.python.org, so I can see the logs and push out a new version. >>>>> I haven't needed to push a version out before, and I'm a little hesitant incase I do it wrong and break something. >>>>> >>>>> I also don't want to push code to GAE from my fork, until my PR gets accepted or else someone else in the future might deploy the original one again and remove my fix. >>>>> >>>>> Two things: >>>>> >>>>> 1. Now that we have the pypi CDN up, do we still need this mirror? >>>> >>>> Honestly probably not. Mirrors are less important from a availability/speed side of things now and will likely move to being more useful for companies and such to use. >>>> >>>> OK, what would be the procedure for removing a mirror? Anyone know who is in charge of this mirror? I think Guido had it setup when he worked at Google, and google is paying for the costs of the mirror, but now that he doesn't work for Google, not sure who might be the contact person on that side. >>>> >>>> I've asked Guido who are admins on the account it to get it turned off. If he doesn't know I can try to find out internally. >>>> >>>> >>>> Brett, >>>> >>>> Here is the owner list that I can see on GAE >>>> >>>> - guido >>>> - kencochrane (me) >>>> - kumar.mcmillan >>>> - martin.v.loewis >>>> - r1chardj0n3s >>>> >>>> So Guido didn't even know he was an owner. =) >>>> >>>> In terms of shutting down the app, you will want to do two things. First is empty out the cron.yaml file; it should have nothing more than "cron:". After that you probably want to return 404 for everything; see http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. >>>> >>>> If you want, Ken, I can clone https://bitbucket.org/kencochrane/pypi-appengine and send you a pull request to do all of this since I recently did something similar for py3ksupport.appspot.com when I shut it down. >>>> >>>> Brett, >>>> >>>> If our goal is to shut it down, then yes please, if you can send a pull request that would be great.. We should also probably remove it from the pypi mirror pool before we do this so it no longer gets traffic sent it's way. >>>> >>>> >>>> >>>> >>>> >>>> If we can get it up to date again, I think it is fine, but an out of date mirror is not useful to anyone, and it could cause problems in the long run. >>>> >>>>> >>>>> 2. If yes to 1. if someone can take a minute to review my PR, and leave comments, or if you have the power, accept my pull request and push out a new version so we can get the mirror up to date. >>>> >>>> I don't have such permission sadly. >>>> >>>> >>>> Thank you anyway. >>> >>> Paging Noah to kill the DNS Unless there's any objections to removing it? >> >> If no one complains in the next 24-ish hours I'll just point it back at pypi.python.org like we did with d.pypi. > > This is now complete. In another 24 hours there should be no traffic to the GAE app and it can just be archived or deleted. Thank you! From julian at grayvines.com Sun Jun 9 20:46:22 2013 From: julian at grayvines.com (Julian Berman) Date: Sun, 9 Jun 2013 14:46:22 -0400 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: <05c820d2d5984850b6dd8543ae3af5f9@BLUPR06MB003.namprd06.prod.outlook.com> References: <05c820d2d5984850b6dd8543ae3af5f9@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: Hey Jason. I was getting it before you removed it I think (and yeah, still get it now if I `pip install https://bitbucket.org/pypa/setuptools/downloads/distribute-0.7.zip`). An example of such a package is pudb, if you'd like to try and reproduce (or I can open an issue on the tracker, just wasn't sure I wasn't missing something), but even if I install with easy_install I still seem to get the same output: ? easy_install pudb Julian at air Searching for pudb Reading https://pypi.python.org/simple/pudb/ Best match: pudb 2013.2 Downloading https://pypi.python.org/packages/source/p/pudb/pudb-2013.2.tar.gz#md5=16411aaffc6f7ae63aba3830ba4ebee5 Processing pudb-2013.2.tar.gz Writing /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/setup.cfg Running pudb-2013.2/setup.py -q bdist_egg --dist-dir /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/egg-dist-tmp-6RBU6H Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35 Building a Distribute egg in /private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2 Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /usr/local/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/distribute-0.6.35-py2.7.egg error: None ~ ? pip freeze | grep distribute Julian at air distribute==0.7 On Sun, Jun 9, 2013 at 2:29 PM, Jason R. Coombs wrote: > Hi Julian,**** > > I believe for those packages that call `use_setuptools()` > from `distribute_setup.py`, the only option to support them is to install > Distribute 0.7 (the stub package), which I believe will satisfy the > use_setuptools() call so it doesn?t attempt to install Distribute 0.6.**** > > ** ** > > As mentioned in another post, I?ve temporarily pulled > Distribute 0.7 from PyPI until we can determine a way to avoid the pip > upgrade issue. In the meantime, if you install Distribute 0.7 from > https://bitbucket.org/pypa/setuptools/downloads, does that solve your > issue?**** > > ** ** > > Regards,**** > > Jason**** > > ** ** > > *From:* Distutils-SIG [mailto:distutils-sig-bounces+jaraco= > jaraco.com at python.org] *On Behalf Of *Julian Berman > *Sent:* Sunday, 09 June, 2013 14:21 > *To:* distutils-sig at python.org > *Subject:* Re: [Distutils] Setuptools/Distribute error with 0.7.2**** > > ** ** > > (I was also getting this until I used easy_install to upgrade)**** > > ** ** > > What is supposed to happen with stuff that used to do `use_setuptools()` > in their setup.py now?**** > > ** ** > > I upgraded to setuptools 0.7.2, but now if I try to install a thing that > has that in the setup.py, it seems to want to try and install distribute, > and I get:**** > > ** ** > > Downloading/unpacking pudb (from -r > /Users/Julian/.local/share/virtualenvs/requirements-every-virtualenv.txt > (line 2))**** > > Using download cache from /Users/Julian/Library/Caches/pip/https%3A%2F% > 2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpudb%2Fpudb-2013.2.tar.gz**** > > Running setup.py egg_info for package pudb**** > > Downloading > http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz > **** > > Extracting in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3**** > > Now working in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 > **** > > Building a Distribute egg in > /Users/Julian/.local/share/virtualenvs/great/build/pudb**** > > Traceback (most recent call last):**** > > File "setup.py", line 45, in **** > > exec(init_file.read(), d)**** > > File "", line 8, in **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", > line 2, in **** > > from setuptools.extension import Extension, Library**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", > line 5, in **** > > from setuptools.dist import _get_unpatched**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", > line 6, in **** > > from setuptools.command.install import install**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", > line 8, in **** > > from setuptools.command import install_scripts**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", > line 3, in **** > > from pkg_resources import Distribution, PathMetadata, > ensure_directory**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > add_activation_listener(lambda dist: dist.activate())**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 710, in subscribe**** > > callback(dist)**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > add_activation_listener(lambda dist: dist.activate())**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2257, in activate**** > > self.insert_on(path)**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2358, in insert_on**** > > "with distribute. Found one at %s" % str(self.location))**** > > ValueError: A 0.7-series setuptools cannot be installed with > distribute. Found one at > /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg > **** > > > /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg > **** > > Traceback (most recent call last):**** > > File "", line 16, in **** > > File > "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, > in **** > > use_setuptools()**** > > File "distribute_setup.py", line 152, in use_setuptools**** > > return _do_download(version, download_base, to_dir, download_delay) > **** > > File "distribute_setup.py", line 132, in _do_download**** > > _build_egg(egg, tarball, to_dir)**** > > File "distribute_setup.py", line 123, in _build_egg**** > > raise IOError('Could not build the egg.')**** > > IOError: Could not build the egg.**** > > Complete output from command python setup.py egg_info:**** > > Downloading > http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz > **** > > ** ** > > Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3** > ** > > ** ** > > Now working in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 > **** > > ** ** > > Building a Distribute egg in > /Users/Julian/.local/share/virtualenvs/great/build/pudb**** > > ** ** > > Traceback (most recent call last):**** > > ** ** > > File "setup.py", line 45, in **** > > ** ** > > exec(init_file.read(), d)**** > > ** ** > > File "", line 8, in **** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", > line 2, in **** > > ** ** > > from setuptools.extension import Extension, Library**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", > line 5, in **** > > ** ** > > from setuptools.dist import _get_unpatched**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", > line 6, in **** > > ** ** > > from setuptools.command.install import install**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", > line 8, in **** > > ** ** > > from setuptools.command import install_scripts**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", > line 3, in **** > > ** ** > > from pkg_resources import Distribution, PathMetadata, ensure_directory > **** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > ** ** > > add_activation_listener(lambda dist: dist.activate())**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 710, in subscribe**** > > ** ** > > callback(dist)**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > ** ** > > add_activation_listener(lambda dist: dist.activate())**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2257, in activate**** > > ** ** > > self.insert_on(path)**** > > ** ** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2358, in insert_on**** > > ** ** > > "with distribute. Found one at %s" % str(self.location))**** > > ** ** > > ValueError: A 0.7-series setuptools cannot be installed with distribute. > Found one at > /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg > **** > > ** ** > > > /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg > **** > > ** ** > > Traceback (most recent call last):**** > > ** ** > > File "", line 16, in **** > > ** ** > > File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", > line 5, in **** > > ** ** > > use_setuptools()**** > > ** ** > > File "distribute_setup.py", line 152, in use_setuptools**** > > ** ** > > return _do_download(version, download_base, to_dir, download_delay)*** > * > > ** ** > > File "distribute_setup.py", line 132, in _do_download**** > > ** ** > > _build_egg(egg, tarball, to_dir)**** > > ** ** > > File "distribute_setup.py", line 123, in _build_egg**** > > ** ** > > raise IOError('Could not build the egg.')**** > > ** ** > > IOError: Could not build the egg.**** > > ** ** > > ----------------------------------------**** > > Command python setup.py egg_info failed with error code 1 in > /Users/Julian/.local/share/virtualenvs/great/build/pudb**** > > S**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Jun 9 22:43:10 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 9 Jun 2013 13:43:10 -0700 Subject: [Distutils] PEP439 and upgrading pip Message-ID: maybe this was covered before, but I'm looking for clarification on how pip upgrades are to work under PEP439. I see 3 relevant bits right now: 1) "attempts to import pip machinery. If it can then the pip command proceeds as normal" i.e., once it has a pip installed, it doesn't automatically upgrade on subsequent imports. 2) "A bootstrap is used in the place of a the full pip code so that we don't have to bundle pip and also the install tool is upgradeable outside of the regular Python upgrade timeframe and processes" to be clear, what is the "install tool" here? pip? or the bootstrap? 3) "Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip3 install . This installation will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files."" I guess this means "manual upgrades" as well? this special logic is to be inside pip itself? or an override of "install" in the bootstrap (that detects when it's a pip wheel)? Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Sun Jun 9 22:57:20 2013 From: aclark at aclark.net (Alex Clark) Date: Sun, 9 Jun 2013 20:57:20 +0000 (UTC) Subject: [Distutils] Open source community relations: Was: Re: PyPI Download Counts References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> Message-ID: Noah Kantrowitz coderanger.net> writes: > > There is another way, make awesome and wait for history to determine who was happy :) Tee hee hee! :-) OK this is my 3rd (and likely last) attempt to steer this thread away from the specific, and contemplate the broader implications. The PyPI CDN is awesome (seriously, I appreciate all the hard work and progress from Donald, Noah, Fastly, et al, as I've said recently in another thread). And I'm prepared to be the only one interested in discussing the broader implications, at which point I'll go OT and shut up. But FWIW I bother to comment here for a reason: to learn from this incident and apply that knowledge to future incidents, and to help others do the same. Alex > > --Noah > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From qwcode at gmail.com Sun Jun 9 23:54:05 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 9 Jun 2013 14:54:05 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <51B4BA23.2060009@numenet.com> Message-ID: > It was only working in python2 before now anyway. ( > https://github.com/pypa/pip/issues/650) > pip is fundamentally dependent on setuptools to perform upgrades. > with distribute-0.6.X upgrades, even though pip uninstalled distribute as part of the upgrade, it ran the install subprocess with the cwd equal to the build dir, so it could import setuptools from the unpacked download. but now, distribute-0.7 is just a shell, that contains no importable setuptools that pip can use. we could *make* it work I guess, by enforcing non-standard logic when dealing with distribute (i.e. get setuptools-0.7 installed first; by default, setuptools gets queued to be installed after the distribute upgrade) but like I said before, it's much more motivating to think about PEP439, than this dreary setuptools/distribute legacy headache stuff. Marcus > the best fix I think is to move faster on making pip "PEP439 compliant" > - i.e. have pip be able to at least install from wheels w/o needing > setuptools, which would remove the bootstrap headache > - see https://github.com/pypa/pip/issues/863 > - this could likely involve pip replacing it's use of pkg_resources > with distlib (https://github.com/pypa/pip/pull/909) > > Marcus > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdouche at gmail.com Mon Jun 10 00:03:37 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Mon, 10 Jun 2013 00:03:37 +0200 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: On Sun, Jun 9, 2013 at 7:49 PM, Jason R. Coombs wrote: > I'll put ez_setup.py for 0.7.2 in downloads for now. You can assume that's > where the permalink will be. The Python community works hard to make PyPI a beautiful place, without external url. And your first "public" action is to put the most important file on BitBucket. Seriously? > If anyone has a better suggestion, please raise it. http://pypi.python.org/simple/setuptools/ez_setup.py Advantages: - Mirrored (don't forget the internal mirrors) with a PyPI tool - No dependency on a private infrastructure - Only one rule on Firewall (pass on pypi.python.org) - Works if Bitbucket is down Disadvantages: - ? -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From chris at simplistix.co.uk Mon Jun 10 00:39:29 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 09 Jun 2013 23:39:29 +0100 Subject: [Distutils] Permission Denied errors on Windows 7 with buildout 1.7.1 Message-ID: <51B50421.2020107@simplistix.co.uk> Hi All, I'm seeing this exception when running a particular buildout on Windows: C:\Jenkins\workspace\checker-buildout\aeb5917b>bin\buildout Develop: 'C:\\Jenkins\\workspace\\checker-buildout\\aeb5917b\\.' Installing py. Generated script 'C:\\Jenkins\\workspace\\checker-buildout\\aeb5917b\\bin\\checker'. Generated script 'C:\\Jenkins\\workspace\\checker-buildout\\aeb5917b\\bin\\tox'. Generated script 'C:\\Jenkins\\workspace\\checker-buildout\\aeb5917b\\bin\\tox-quickstart'. While: Installing py. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "c:\users\jenkins\.buildout\eggs\zc.buildout-1.7.1-py2.7.egg\zc\buildout\buildout.py", line 1866, in main getattr(buildout, command)(args) File "c:\users\jenkins\.buildout\eggs\zc.buildout-1.7.1-py2.7.egg\zc\buildout\buildout.py", line 625, in install installed_files = self[part]._call(recipe.install) File "c:\users\jenkins\.buildout\eggs\zc.buildout-1.7.1-py2.7.egg\zc\buildout\buildout.py", line 1345, in _call return f() File "c:\users\jenkins\.buildout\eggs\zc.recipe.egg-1.3.2-py2.7.egg\zc\recipe\egg\egg.py", line 173, in install return self._install(reqs, ws, scripts) File "c:\users\jenkins\.buildout\eggs\zc.recipe.egg-1.3.2-py2.7.egg\zc\recipe\egg\egg.py", line 195, in _install relative_paths=self._relative_paths File "c:\users\jenkins\.buildout\eggs\zc.buildout-1.7.1-py2.7.egg\zc\buildout\easy_install.py", line 1223, in scripts initialization, executable, arguments) File "c:\users\jenkins\.buildout\eggs\zc.buildout-1.7.1-py2.7.egg\zc\buildout\easy_install.py", line 1350, in _generate_scripts module_name, attrs, arguments, block_site=block_site)) File "c:\users\jenkins\.buildout\eggs\zc.buildout-1.7.1-py2.7.egg\zc\buildout\easy_install.py", line 1498, in _script return _write_script(dest, contents, 'script') File "c:\users\jenkins\.buildout\eggs\zc.buildout-1.7.1-py2.7.egg\zc\buildout\easy_install.py", line 1463, in _write_script open(exe, 'wb').write(new_data) IOError: [Errno 13] Permission denied: 'C:\\Jenkins\\workspace\\checker-buildout\\aeb5917b\\bin\\buildout.exe' This happens every time, on Python 2.5, 2.6 and 2.7. I've tried blowing the workspace away, makes no difference: http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.7,label=windows/console http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.6,label=windows/console http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.5,label=windows/console Any idea what's going on here and how I can fix it? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Jun 10 00:43:38 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 09 Jun 2013 23:43:38 +0100 Subject: [Distutils] Permission Denied errors on Windows 7 with buildout 1.7.1 In-Reply-To: <51B50421.2020107@simplistix.co.uk> References: <51B50421.2020107@simplistix.co.uk> Message-ID: <51B5051A.4060202@simplistix.co.uk> On 09/06/2013 23:39, Chris Withers wrote: > This happens every time, on Python 2.5, 2.6 and 2.7. I've tried blowing > the workspace away, makes no difference: > > http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.7,label=windows/console > > http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.6,label=windows/console > > http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.5,label=windows/console Forgot to say, the buildout in question is here: https://github.com/Simplistix/checker/blob/master/buildout.cfg cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From tseaver at palladion.com Mon Jun 10 00:53:31 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 09 Jun 2013 18:53:31 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: <51B5076B.4090405@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/2013 01:49 PM, Jason R. Coombs wrote: > Is the bitbucket 'raw' source not suitable? It uses the version tag to > indicate precisely which file is appropriate, which means we don't > have to redundantly host that file elsewhere. > > Oh, I think I see - there's no link that doesn't change - you want a > permalink for the latest stable release. I hadn't considered that use > case, but you're right - we do want that. Perhaps the file could be > added to pythonhosted.org alongside the documentation. Or maybe it > belongs on bitbucket in `downloads`. > > I need to figure out which is more straightforward w.r.t. > authentication. The distribute process was messy because it involved > SSH and keys that had to be manually managed. > > I'm leaning toward uploading it to BitBucket downloads as part of the > release script. > > I'll put ez_setup.py for 0.7.2 in downloads for now. You can assume > that's where the permalink will be. > > If anyone has a better suggestion, please raise it. I would prefer a URL on a host managed by the Python infrastructure team: either PyPI or the pythonhosted org. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG1B2sACgkQ+gerLs4ltQ7H8gCdE/g1UBfr74s7v5tmuHLXdktx 3L0AoMxPG48F5KM8Ae+v5A1g9nU2B4To =pTON -----END PGP SIGNATURE----- From eric at trueblade.com Mon Jun 10 01:19:04 2013 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 09 Jun 2013 19:19:04 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: <51B5076B.4090405@palladion.com> References: <51B5076B.4090405@palladion.com> Message-ID: <51B50D68.3050704@trueblade.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 6/9/2013 6:53 PM, Tres Seaver wrote: > On 06/09/2013 01:49 PM, Jason R. Coombs wrote: >> Is the bitbucket 'raw' source not suitable? It uses the version >> tag to indicate precisely which file is appropriate, which means >> we don't have to redundantly host that file elsewhere. > >> Oh, I think I see - there's no link that doesn't change - you >> want a permalink for the latest stable release. I hadn't >> considered that use case, but you're right - we do want that. >> Perhaps the file could be added to pythonhosted.org alongside the >> documentation. Or maybe it belongs on bitbucket in `downloads`. > >> I need to figure out which is more straightforward w.r.t. >> authentication. The distribute process was messy because it >> involved SSH and keys that had to be manually managed. > >> I'm leaning toward uploading it to BitBucket downloads as part of >> the release script. > >> I'll put ez_setup.py for 0.7.2 in downloads for now. You can >> assume that's where the permalink will be. > >> If anyone has a better suggestion, please raise it. > > I would prefer a URL on a host managed by the Python infrastructure > team: either PyPI or the pythonhosted org. And benefiting from the PyPI CDN would be great. - -- Eric. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJRtQ1oAAoJENxauZFcKtNxHVsH/1AXswEmeQAjH/GuRqOmNoQH uDIE6D/y3wrld0fyzejpKqponaaiiChvwqfFCL+TU47qsScOl9eyBAoylgeCvVoR cAmeDlyrn52RILKE9F5p9BhAeFTtWdi+87LIvv3BWu0y/HdrWiYoCMA94xyolSF9 rLPb4hJuVlsau3+jcxi5iNtlTMRsyL1wYrygAmnp7tcJ3LiQ23SQI9kxhdGXvunY vQvzptMEZOLktyX+5A4IPiFXzsuPkLY5RYD7F+7B/Sn6oH8TSNLEbmagP4U3C/bX 7TAMT4MDTs2tzcby/lKwveQeI7+3VIBrJ1bHhLgqvIBccwTe4iXV4o1lcgGDXgo= =3NIb -----END PGP SIGNATURE----- From jaraco at jaraco.com Mon Jun 10 03:37:33 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Mon, 10 Jun 2013 01:37:33 +0000 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <51B4BA23.2060009@numenet.com> Message-ID: <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> I think it would be highly desirable to add support for pip to handle the upgrade from distribute 0.6 to 0.7. You'll note that because 0.7 depends on setuptools 0.7 that 0.7 has already been downloaded. Perhaps a shim like you propose would work, or it's also conceivable that distribute 0.7 could include a setuptools 0.7 source tree which pip could leverage (but not install). Yes, PEP 439 will be awesome, but I think this important milestone for setuptools is also essential to simplify the landscape in order to help migrate users in a sustainable way to the new tools, so we've got to find a way to make it happen for the good of the eco system. From: qwelby at gmail.com [mailto:qwelby at gmail.com] On Behalf Of Marcus Smith Sent: Sunday, 09 June, 2013 17:54 To: Jason R. Coombs Cc: Liam Kirsher; distutils-sig at python.org Subject: Re: [Distutils] Setuptools/Distribute error with 0.7.2 It was only working in python2 before now anyway. (https://github.com/pypa/pip/issues/650) pip is fundamentally dependent on setuptools to perform upgrades. with distribute-0.6.X upgrades, even though pip uninstalled distribute as part of the upgrade, it ran the install subprocess with the cwd equal to the build dir, so it could import setuptools from the unpacked download. but now, distribute-0.7 is just a shell, that contains no importable setuptools that pip can use. we could *make* it work I guess, by enforcing non-standard logic when dealing with distribute (i.e. get setuptools-0.7 installed first; by default, setuptools gets queued to be installed after the distribute upgrade) but like I said before, it's much more motivating to think about PEP439, than this dreary setuptools/distribute legacy headache stuff. Marcus the best fix I think is to move faster on making pip "PEP439 compliant" - i.e. have pip be able to at least install from wheels w/o needing setuptools, which would remove the bootstrap headache - see https://github.com/pypa/pip/issues/863 - this could likely involve pip replacing it's use of pkg_resources with distlib (https://github.com/pypa/pip/pull/909) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From ncoghlan at gmail.com Mon Jun 10 03:35:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Jun 2013 11:35:01 +1000 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> Message-ID: On 10 June 2013 06:04, Alex Clark wrote: > Donald Stufft stufft.io> writes: > >> So yes. I broke Download counts because they were not more important than >> people being able to actually use PyPI to install from. > > > FWIW: > > > You missed the moral of the story: when you make a decision like this, > someone will *always* disagree with you (even over the most trivial things). > And even if they don't, they may disagree with your approach (e.g. why not > sort problems with download counts before enabling the CDN) So the only way > to make everyone happy is to consider everyone who will be affected by your > actions, before you take action. And, indeed, we plan to run future changes of this magnitude through the PEP process for exactly this reason. We can't promise not to break some features in order to achieve gains we think are worth the loss, but we *can* promise not to break such features without advance warning and explicit consideration of alternatives that may allow us to avoid the breakage in the first place. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jun 10 03:43:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Jun 2013 11:43:05 +1000 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> References: <51B4BA23.2060009@numenet.com> <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On 10 June 2013 11:37, Jason R. Coombs wrote: > I think it would be highly desirable to add support for pip to handle the > upgrade from distribute 0.6 to 0.7. You?ll note that because 0.7 depends on > setuptools 0.7 that 0.7 has already been downloaded. Perhaps a shim like you > propose would work, or it?s also conceivable that distribute 0.7 could > include a setuptools 0.7 source tree which pip could leverage (but not > install). > > Yes, PEP 439 will be awesome, but I think this important milestone for > setuptools is also essential to simplify the landscape in order to help > migrate users in a sustainable way to the new tools, so we?ve got to find a > way to make it happen for the good of the eco system. I believe Marcus's point was that if pip is using a vendored copy of distlib to do the installs (rather than relying on a separately installed instance of setuptools), then it would be able to upgrade/downgrade setuptools and distribute in both Python 2 and Python 3 without any dependency issues. On the other hand, a near-term special casing of distribute to force installing/upgrading setuptools 0.7 before upgrading distribute to 0.7 is likely a lower impact change to pip. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jun 10 03:51:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Jun 2013 11:51:49 +1000 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: On 10 June 2013 03:49, Jason R. Coombs wrote: > I'm leaning toward uploading it to BitBucket downloads as part of the release > script. > > I'll put ez_setup.py for 0.7.2 in downloads for now. You can assume that's > where the permalink will be. I think that's a good way to publish it securely to the PSF infrastructure team (at least for now) > If anyone has a better suggestion, please raise it. As others have suggested, we should work with Donald and Noah to get the bootstrapping script an official home somewhere on pypi.python.org. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From qwcode at gmail.com Mon Jun 10 04:02:58 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 9 Jun 2013 19:02:58 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> References: <51B4BA23.2060009@numenet.com> <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: agreed, a quick fix for the upcoming pip-1.4 would be nice. (PEP439 is 1.5 at best) here's the pip issue for this problem (set for 1.4) https://github.com/pypa/pip/issues/986 I'll discuss options in the issue. Marcus On Sun, Jun 9, 2013 at 6:37 PM, Jason R. Coombs wrote: > I think it would be highly desirable to add support for pip to handle the > upgrade from distribute 0.6 to 0.7. You?ll note that because 0.7 depends on > setuptools 0.7 that 0.7 has already been downloaded. Perhaps a shim like > you propose would work, or it?s also conceivable that distribute 0.7 could > include a setuptools 0.7 source tree which pip could leverage (but not > install).**** > > ** ** > > Yes, PEP 439 will be awesome, but I think this important milestone for > setuptools is also essential to simplify the landscape in order to help > migrate users in a sustainable way to the new tools, so we?ve got to find a > way to make it happen for the good of the eco system.**** > > ** ** > > *From:* qwelby at gmail.com [mailto:qwelby at gmail.com] *On Behalf Of *Marcus > Smith > *Sent:* Sunday, 09 June, 2013 17:54 > *To:* Jason R. Coombs > *Cc:* Liam Kirsher; distutils-sig at python.org > *Subject:* Re: [Distutils] Setuptools/Distribute error with 0.7.2**** > > ** ** > > ** ** > > It was only working in python2 before now anyway. ( > https://github.com/pypa/pip/issues/650)**** > > pip is fundamentally dependent on setuptools to perform upgrades.**** > > ** ** > > with distribute-0.6.X upgrades, even though pip uninstalled distribute as > part of the upgrade, it ran the install subprocess with the cwd equal to > the build dir, so it could import setuptools from the unpacked download.** > ** > > but now, distribute-0.7 is just a shell, that contains no importable > setuptools that pip can use.**** > > we could *make* it work I guess, by enforcing non-standard logic when > dealing with distribute (i.e. get setuptools-0.7 installed first; by > default, setuptools gets queued to be installed after the distribute > upgrade)**** > > but like I said before, it's much more motivating to think about PEP439, > than this dreary setuptools/distribute legacy headache stuff.**** > > ** ** > > Marcus**** > > ** ** > > **** > > the best fix I think is to move faster on making pip "PEP439 compliant"*** > * > > - i.e. have pip be able to at least install from wheels w/o needing > setuptools, which would remove the bootstrap headache**** > > - see https://github.com/pypa/pip/issues/863**** > > - this could likely involve pip replacing it's use of pkg_resources > with distlib (https://github.com/pypa/pip/pull/909)**** > > ** ** > > Marcus **** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 10 04:28:59 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 9 Jun 2013 22:28:59 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> On Jun 9, 2013, at 9:51 PM, Nick Coghlan wrote: > On 10 June 2013 03:49, Jason R. Coombs wrote: >> I'm leaning toward uploading it to BitBucket downloads as part of the release >> script. >> >> I'll put ez_setup.py for 0.7.2 in downloads for now. You can assume that's >> where the permalink will be. > > I think that's a good way to publish it securely to the PSF > infrastructure team (at least for now) > >> If anyone has a better suggestion, please raise it. > > As others have suggested, we should work with Donald and Noah to get > the bootstrapping script an official home somewhere on > pypi.python.org. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig If you want it under pypi.python.org it'll probably need manually sent to myself for the time being. I believe it could easily be hosted via python hosted.org though without needing manual intervention. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tseaver at palladion.com Mon Jun 10 04:41:51 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 09 Jun 2013 22:41:51 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> References: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/2013 10:28 PM, Donald Stufft wrote: > > On Jun 9, 2013, at 9:51 PM, Nick Coghlan wrote: > >> On 10 June 2013 03:49, Jason R. Coombs wrote: >>> I'm leaning toward uploading it to BitBucket downloads as part of >>> the release script. >>> >>> I'll put ez_setup.py for 0.7.2 in downloads for now. You can >>> assume that's where the permalink will be. >> >> I think that's a good way to publish it securely to the PSF >> infrastructure team (at least for now) >> >>> If anyone has a better suggestion, please raise it. >> >> As others have suggested, we should work with Donald and Noah to >> get the bootstrapping script an official home somewhere on >> pypi.python.org. >> > If you want it under pypi.python.org it'll probably need manually > sent to myself for the time being. I believe it could easily be hosted > via python hosted.org though without needing manual intervention. I'm not wedded to anyplace in particular. Having the bootstrap needed to make effective use of PyPI hosted on PyPI itself makes a lot of sense to me. Maybe under a URL like this one? https://pypi.python.org/bootstraps/ez_setup.py The main key is that the URL should be immutable (so we can doucment it widely, and use it in scripts) and have availability as good as the cheeseshop (I'm not knocking Bitbucket in particular). If the file were copied into the Sphinx docs uploaded to pythonhosted.org, that would probably work just as well. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG1PO8ACgkQ+gerLs4ltQ64WQCeO0tVhPBjlmzvtzdBx0C4V5Hr lVAAniVJ4XuAnPfV7NXT7KvcextrrTaJ =ILKd -----END PGP SIGNATURE----- From regebro at gmail.com Mon Jun 10 08:21:21 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 10 Jun 2013 08:21:21 +0200 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> References: <51B4BA23.2060009@numenet.com> <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On Mon, Jun 10, 2013 at 3:37 AM, Jason R. Coombs wrote: > I think it would be highly desirable to add support for pip to handle the > upgrade from distribute 0.6 to 0.7. You?ll note that because 0.7 depends on > setuptools 0.7 that 0.7 has already been downloaded. Perhaps a shim like you > propose would work, or it?s also conceivable that distribute 0.7 could > include a setuptools 0.7 source tree which pip could leverage (but not > install). Wild idea: Distribute 0.7 includes a non 2to3 source tree that doesn't get installed. This would also solve the upgrade issue under Python 3, and serve as an early test for getting rid of 2to3. :-) That might then mean that Distribute 0.7 would end up only be pip ugradeable on Python 2.6 and later, but maybe that's OK? I certainly don't mind. :-) //Lennart From holger at merlinux.eu Mon Jun 10 09:34:32 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 10 Jun 2013 07:34:32 +0000 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> Message-ID: <20130610073432.GK7789@merlinux.eu> On Mon, Jun 10, 2013 at 11:35 +1000, Nick Coghlan wrote: > On 10 June 2013 06:04, Alex Clark wrote: > > Donald Stufft stufft.io> writes: > > > >> So yes. I broke Download counts because they were not more important than > >> people being able to actually use PyPI to install from. > > > > > > FWIW: > > > > > > You missed the moral of the story: when you make a decision like this, > > someone will *always* disagree with you (even over the most trivial things). > > And even if they don't, they may disagree with your approach (e.g. why not > > sort problems with download counts before enabling the CDN) So the only way > > to make everyone happy is to consider everyone who will be affected by your > > actions, before you take action. > > And, indeed, we plan to run future changes of this magnitude through > the PEP process for exactly this reason. We can't promise not to break > some features in order to achieve gains we think are worth the loss, > but we *can* promise not to break such features without advance > warning and explicit consideration of alternatives that may allow us > to avoid the breakage in the first place. Nick, i welcome this intention but could you now and in the future be explicit about who "we" is? It's not even obvious to me and i have written a recent PEP in this area and am following most of the mails here for a while, also participating in some actions behind the scene. thanks, holger From ncoghlan at gmail.com Mon Jun 10 09:51:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Jun 2013 17:51:01 +1000 Subject: [Distutils] PyPI Download Counts In-Reply-To: <20130610073432.GK7789@merlinux.eu> References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> <20130610073432.GK7789@merlinux.eu> Message-ID: On 10 June 2013 17:34, holger krekel wrote: > On Mon, Jun 10, 2013 at 11:35 +1000, Nick Coghlan wrote: >> On 10 June 2013 06:04, Alex Clark wrote: >> > Donald Stufft stufft.io> writes: >> > >> >> So yes. I broke Download counts because they were not more important than >> >> people being able to actually use PyPI to install from. >> > >> > >> > FWIW: >> > >> > >> > You missed the moral of the story: when you make a decision like this, >> > someone will *always* disagree with you (even over the most trivial things). >> > And even if they don't, they may disagree with your approach (e.g. why not >> > sort problems with download counts before enabling the CDN) So the only way >> > to make everyone happy is to consider everyone who will be affected by your >> > actions, before you take action. >> >> And, indeed, we plan to run future changes of this magnitude through >> the PEP process for exactly this reason. We can't promise not to break >> some features in order to achieve gains we think are worth the loss, >> but we *can* promise not to break such features without advance >> warning and explicit consideration of alternatives that may allow us >> to avoid the breakage in the first place. > > Nick, i welcome this intention but could you now and in the future > be explicit about who "we" is? It's not even obvious to me > and i have written a recent PEP in this area and am following most of > the mails here for a while, also participating in some actions behind > the scene. In the specific case of PyPI, mainly Donald (as the one doing almost all the development work for the recent PyPI improvements), Richard (as the PyPI development lead) and Noah (as the python.org infrastructure lead). I'm less directly involved in PyPI changes (I'm still working mostly on the next draft of PEP 426 rather than anything more near term), but I'm still involved enough to want to say "we" rather than "they" :) The main issue that arose with the PyPI CDN change was that while it *was* discussed, the discussion happened on infrastructure-sig rather than here, so most PyPI client developers didn't hear about it until after the switch was already flipped. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at python.org Mon Jun 10 11:37:42 2013 From: chris at python.org (Chris Withers) Date: Mon, 10 Jun 2013 10:37:42 +0100 Subject: [Distutils] Permission Denied errors on Windows 7 with buildout 1.7.1 In-Reply-To: <51B5051A.4060202@simplistix.co.uk> References: <51B50421.2020107@simplistix.co.uk> <51B5051A.4060202@simplistix.co.uk> Message-ID: <51B59E66.4060809@python.org> On 09/06/2013 23:43, Chris Withers wrote: > On 09/06/2013 23:39, Chris Withers wrote: >> This happens every time, on Python 2.5, 2.6 and 2.7. I've tried blowing >> the workspace away, makes no difference: >> >> http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.7,label=windows/console >> >> >> http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.6,label=windows/console >> >> >> http://jenkins.simplistix.co.uk/job/checker-buildout/114/PYTHON=2.5,label=windows/console >> > > Forgot to say, the buildout in question is here: > > https://github.com/Simplistix/checker/blob/master/buildout.cfg So, infuriatingly, I can't reproduce this on another Windows machine. I've tried blowing away the egg cache on the original machine, no joy, the error still occurs when the buildout is re-run. I'm stumped as to what the difference is that's making a difference here... Any help gratefully received! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Jun 10 12:13:06 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 10 Jun 2013 11:13:06 +0100 Subject: [Distutils] buildout 2 not compatible with python2.5? Message-ID: <51B5A6B2.4060803@simplistix.co.uk> Hi All, Am I right in thinking that buildout 2 isn't compatible with Python 2.5? http://jenkins.simplistix.co.uk/job/checker-buildout/119/ eg: Building a Distribute egg in /tmp/tmpqeG0PG /tmp/tmpqeG0PG/distribute-0.6.45-py2.5.egg File "build/bdist.linux-i686/egg/zc/buildout/easy_install.py", line 133 current_umask = os.umask(0o022) ^ SyntaxError: invalid syntax ...which is a shame, as it seemed to be doing a bit better at fixing the Windows problems I was having... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Jun 10 12:22:01 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 10 Jun 2013 11:22:01 +0100 Subject: [Distutils] buildout 1 still not managing to pin buildout to 1.x Message-ID: <51B5A8C9.6050508@simplistix.co.uk> Hi All, So, having given up on buildout 2 for this package again (need for 2.5 support) I dutifully put in the bootstrap.py from buildout 1: curl http://downloads.buildout.org/1/bootstrap.py > bootstrap.py python2.5 on my mac didn't work, thanks to a broken system python, so on to 2.6, which appeared to go okay except: buzzkill:checker chris$ bin/nosetests zc.buildout 2 needs distribute, not setuptools. Are you using an outdated bootstrap.py? Make sure you have the latest version downloaded from http://downloads.buildout.org/2/bootstrap.py Great... Wonderful... I thought 1/bootstrap.py was supposed to pin to buildout 1 for you without the need for any other voodoo? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at python.org Mon Jun 10 13:00:36 2013 From: chris at python.org (Chris Withers) Date: Mon, 10 Jun 2013 12:00:36 +0100 Subject: [Distutils] Permission Denied errors on Windows 7 with buildout 1.7.1 In-Reply-To: <51B59E66.4060809@python.org> References: <51B50421.2020107@simplistix.co.uk> <51B5051A.4060202@simplistix.co.uk> <51B59E66.4060809@python.org> Message-ID: <51B5B1D4.8000901@python.org> On 10/06/2013 10:37, Chris Withers wrote: > I've tried blowing away the egg cache on the original machine, no joy, > the error still occurs when the buildout is re-run. > > I'm stumped as to what the difference is that's making a difference here... Well, the failing machine generates a buildout-script.py that looks like this: #!"C:\Python27\python.exe" -S import sys sys.path[0:0] = [ 'C:\\Jenkins\\workspace\\checker-buildout\\aeb5917b\\parts\\buildout', ] import os path = sys.path[0] if os.environ.get('PYTHONPATH'): path = os.pathsep.join([path, os.environ['PYTHONPATH']]) os.environ['BUILDOUT_ORIGINAL_PYTHONPATH'] = os.environ.get('PYTHONPATH', '') os.environ['PYTHONPATH'] = path import site # imports custom buildout-generated site.py import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) However, the identical machine that works generates this instead: #!"C:\Python27\python.exe" import sys sys.path[0:0] = [ 'h:\\checker', 'h:\\checker\\eggs\\tox-1.4.3-py2.7.egg', 'h:\\checker\\eggs\\zope.testing-4.1.2-py2.7.egg', 'h:\\checker\\eggs\\zc.recipe.egg-1.3.2-py2.7.egg', 'h:\\checker\\eggs\\zc.buildout-1.7.1-py2.7.egg', 'h:\\checker\\eggs\\testfixtures-3.0.0-py2.7.egg', 'h:\\checker\\eggs\\nose_cov-1.6-py2.7.egg', 'h:\\checker\\eggs\\nose_fixes-1.3-py2.7.egg', 'h:\\checker\\eggs\\nose-1.3.0-py2.7.egg', 'h:\\checker\\eggs\\manuel-1.7.2-py2.7.egg', 'h:\\checker\\eggs\\mock-1.0.1-py2.7.egg', 'h:\\checker\\eggs\\zope.dottedname-4.0.1-py2.7.egg', 'h:\\checker\\eggs\\mailinglogger-3.7.0-py2.7.egg', 'h:\\checker\\eggs\\execute-1.2-py2.7.egg', 'h:\\checker\\eggs\\argparse-1.2.1-py2.7.egg', 'h:\\checker\\eggs\\py-1.4.14-py2.7.egg', 'h:\\checker\\eggs\\virtualenv-1.9.1-py2.7.egg', 'h:\\checker\\eggs\\zope.interface-4.0.5-py2.7-win32.egg', 'h:\\checker\\eggs\\zope.exceptions-4.0.6-py2.7.egg', 'h:\\checker\\eggs\\setuptools-0.7.2-py2.7.egg', 'h:\\checker\\eggs\\cov_core-1.7-py2.7.egg', 'h:\\checker\\eggs\\six-1.3.0-py2.7.egg', 'h:\\checker\\eggs\\coverage-3.6-py2.7-win32.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) Anyone know what could cause the different scripts to be generated? zc.recipe.egg and zc.buildout versions look identical on both machines... *sigh* Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From lists at informa.tiker.net Mon Jun 10 14:37:30 2013 From: lists at informa.tiker.net (Andreas Kloeckner) Date: Mon, 10 Jun 2013 08:37:30 -0400 Subject: [Distutils] distribute_setup.py in package + setuptools 0.7.2 = package uninstallable Message-ID: <871u8arvjp.fsf@ding.tiker.net> Hi all, I've shipped distribute_setup.py in a number of my packages. If a user's install (be it virtualenv or otherwise) becomes "infected" with setuptools 0.7.2, my packages appear to become uninstallable, with errors like the log below. Run the following script to reproduce: 8< ------------------------------------------------ #! /bin/sh set -e set -x virtualenv --no-setuptools mypy source mypy/bin/activate curl -k https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py | python - curl -k https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python - pip install pudb==2013.1 8< ------------------------------------------------ Andreas ------------------------------------------------------- $ pip install pudb==2013.1 Downloading/unpacking pudb==2013.1 Downloading pudb-2013.1.tar.gz (40kB): 40kB downloaded Running setup.py egg_info for package pudb Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /tmp/tmpkaEQAq Now working in /tmp/tmpkaEQAq/distribute-0.6.35 Building a Distribute egg in /home/andreas/mypy/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /home/andreas/mypy/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /home/andreas/mypy/build/pudb/distribute-0.6.35-py2.7.egg Traceback (most recent call last): File "", line 16, in File "/home/andreas/mypy/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. Complete output from command python setup.py egg_info: Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /tmp/tmpkaEQAq Now working in /tmp/tmpkaEQAq/distribute-0.6.35 Building a Distribute egg in /home/andreas/mypy/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/tmp/tmpkaEQAq/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/tmp/tmpkaEQAq/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /home/andreas/mypy/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /home/andreas/mypy/build/pudb/distribute-0.6.35-py2.7.egg Traceback (most recent call last): File "", line 16, in File "/home/andreas/mypy/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. ---------------------------------------- Command python setup.py egg_info failed with error code 1 in /home/andreas/mypy/build/pudb Storing complete log in /home/andreas/.pip/pip.log From qwcode at gmail.com Mon Jun 10 18:08:26 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 10 Jun 2013 09:08:26 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <51B4BA23.2060009@numenet.com> <4b1e01b720094c7eb6efadba3a5d5f6a@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: we're discussing options over in the pip issue https://github.com/pypa/pip/issues/986 On Sun, Jun 9, 2013 at 11:21 PM, Lennart Regebro wrote: > On Mon, Jun 10, 2013 at 3:37 AM, Jason R. Coombs > wrote: > > I think it would be highly desirable to add support for pip to handle the > > upgrade from distribute 0.6 to 0.7. You?ll note that because 0.7 depends > on > > setuptools 0.7 that 0.7 has already been downloaded. Perhaps a shim like > you > > propose would work, or it?s also conceivable that distribute 0.7 could > > include a setuptools 0.7 source tree which pip could leverage (but not > > install). > > Wild idea: Distribute 0.7 includes a non 2to3 source tree that doesn't > get installed. This would also solve the upgrade issue under Python 3, > and serve as an early test for getting rid of 2to3. :-) > > That might then mean that Distribute 0.7 would end up only be pip > ugradeable on Python 2.6 and later, but maybe that's OK? I certainly > don't mind. :-) > > //Lennart > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Mon Jun 10 19:57:14 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 10 Jun 2013 13:57:14 -0400 Subject: [Distutils] buildout 1 still not managing to pin buildout to 1.x In-Reply-To: <51B5A8C9.6050508@simplistix.co.uk> References: <51B5A8C9.6050508@simplistix.co.uk> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/10/2013 06:22 AM, Chris Withers wrote: > Hi All, > > So, having given up on buildout 2 for this package again (need for 2.5 > support) I dutifully put in the bootstrap.py from buildout 1: > > curl http://downloads.buildout.org/1/bootstrap.py > bootstrap.py > > python2.5 on my mac didn't work, thanks to a broken system python, so > on to 2.6, which appeared to go okay except: > > buzzkill:checker chris$ bin/nosetests zc.buildout 2 needs distribute, > not setuptools. Are you using an outdated bootstrap.py? Make sure > you have the latest version downloaded from > http://downloads.buildout.org/2/bootstrap.py > > Great... Wonderful... I thought 1/bootstrap.py was supposed to pin to > buildout 1 for you without the need for any other voodoo? Works for me: $ mkdir /tmp/withers $ cd /tmp/withers $ echo "[buildout] parts =" > buildout.cfg $ wget http://downloads.buildout.org/1/bootstrap.py ... $ /opt/Python-2.5.x/bin/python bootstrap.py Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-\ 0.6c11-py2.5.egg Creating directory '/tmp/withers/bin'. Creating directory '/tmp/withers/parts'. Creating directory '/tmp/withers/eggs'. Creating directory '/tmp/withers/develop-eggs'. Getting distribution for 'setuptools'. /opt/Python-2.5.5/lib/python2.5/distutils/dist.py:263: \ UserWarning: Unknown distribution option: 'src_root' warnings.warn(msg) Got setuptools 0.7.2. Generated script '/tmp/withers/bin/buildout'. $ bin/buildout $ ls eggs setuptools-0.7.2-py2.5.egg zc.buildout-1.7.1-py2.5.eg Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG2E3oACgkQ+gerLs4ltQ4n0ACgnIJb6c4Wf/3i+sY57FY1BRGa emgAoJdTohJwbWmhGCvhzt/Pbv9AJTGT =Bn58 -----END PGP SIGNATURE----- From jaraco at jaraco.com Mon Jun 10 20:55:45 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Mon, 10 Jun 2013 18:55:45 +0000 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> Message-ID: <82C062DF-6751-41FA-A528-87685558B6D5@jaraco.com> There seems to be a dominant opinion that the file should be on PyPI, and I don't disagree. One issue is that the file must be mutable. It necessarily contains a reference to the preferred version to be downloaded. That version changes over time. It's typically incremented, but sometimes decremented. It's possible the script could be updated to otherwise discover the most appropriate version. As it's currently implemented however, the file must be expected to change over time. Because the file must change during each release, the system needs to be able to accept an updated version. This is why the Bitbucket Link was referenced, because it can include that version information. I'm open to suggestions on how we can create a perma link to a file that changes over time on PyPI. For now, the most authoritative place I could think of for that file was the bit bucket downloads. Let's identify a mechanism to do the same on PyPI and get that into set of tools for the next release. Sent from my comm On 2013-06-09, at 22:42, "Tres Seaver" wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/09/2013 10:28 PM, Donald Stufft wrote: >> >> On Jun 9, 2013, at 9:51 PM, Nick Coghlan wrote: >> >>> On 10 June 2013 03:49, Jason R. Coombs wrote: >>>> I'm leaning toward uploading it to BitBucket downloads as part of >>>> the release script. >>>> >>>> I'll put ez_setup.py for 0.7.2 in downloads for now. You can >>>> assume that's where the permalink will be. >>> >>> I think that's a good way to publish it securely to the PSF >>> infrastructure team (at least for now) >>> >>>> If anyone has a better suggestion, please raise it. >>> >>> As others have suggested, we should work with Donald and Noah to >>> get the bootstrapping script an official home somewhere on >>> pypi.python.org. >> If you want it under pypi.python.org it'll probably need manually >> sent to myself for the time being. I believe it could easily be hosted >> via python hosted.org though without needing manual intervention. > > I'm not wedded to anyplace in particular. Having the bootstrap needed to > make effective use of PyPI hosted on PyPI itself makes a lot of sense to > me. Maybe under a URL like this one? > > https://pypi.python.org/bootstraps/ez_setup.py > > The main key is that the URL should be immutable (so we can doucment it > widely, and use it in scripts) and have availability as good as the > cheeseshop (I'm not knocking Bitbucket in particular). > > If the file were copied into the Sphinx docs uploaded to > pythonhosted.org, that would probably work just as well. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlG1PO8ACgkQ+gerLs4ltQ64WQCeO0tVhPBjlmzvtzdBx0C4V5Hr > lVAAniVJ4XuAnPfV7NXT7KvcextrrTaJ > =ILKd > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2899 bytes Desc: not available URL: From donald at stufft.io Mon Jun 10 21:14:38 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 10 Jun 2013 15:14:38 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: <82C062DF-6751-41FA-A528-87685558B6D5@jaraco.com> References: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> <82C062DF-6751-41FA-A528-87685558B6D5@jaraco.com> Message-ID: <3F569970-3D95-4111-92DC-2042DFBC111B@stufft.io> On Jun 10, 2013, at 2:55 PM, "Jason R. Coombs" wrote: > There seems to be a dominant opinion that the file should be on PyPI, and I don't disagree. > > One issue is that the file must be mutable. It necessarily contains a reference to the preferred version to be downloaded. That version changes over time. It's typically incremented, but sometimes decremented. > > It's possible the script could be updated to otherwise discover the most appropriate version. As it's currently implemented however, the file must be expected to change over time. > > Because the file must change during each release, the system needs to be able to accept an updated version. This is why the Bitbucket Link was referenced, because it can include that version information. > > I'm open to suggestions on how we can create a perma link to a file that changes over time on PyPI. Put a version in the filename/path, and have a "latest" tag that just 301 redirects to whatever is latest. /bootstrap/setuptools/latest/ez_setup.py -> 301 Redirect to /bootstrap/setuptools/0.7.2/ez_setup.py > > For now, the most authoritative place I could think of for that file was the bit bucket downloads. Let's identify a mechanism to do the same on PyPI and get that into set of tools for the next release. > > Sent from my comm ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Mon Jun 10 23:53:25 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 10 Jun 2013 21:53:25 +0000 Subject: [Distutils] devpi-0.9: pypi caching server and packaging workflow meta tool Message-ID: <20130610215324.GL7789@merlinux.eu> Hi all, i just released devpi-server-0.9, devpi-client-0.9 and a convenience meta "devpi-0.9" package to install them both. Main news for devpi-server, the caching pypi server: - github-style user and index management: you can upload to USER/NAME indexes with the default being "root/dev", an overlay index that serves all pypi.python.org packages and the ones you upload to it. - integrates now well with latest CDN changes, maintains a consistent quite real-time local pypi cache - redis dependency is gone, only pure Python deps left Main news for the initial release of devpi-client, the packaging workflow tool: - provides install/upload/test sub commands to perform common tasks with an index - provides user/index/use/server commands to manage context and the automated server Lastly, to get started super-quickly, there is a new site with all the docs: http://doc.devpi.net I'd like to use the 0.9 series to refine the command line interface, fix bugs and implement some more functionality before heading for a more stable 1.0 release. have fun, holger krekel From p.f.moore at gmail.com Tue Jun 11 10:46:16 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 11 Jun 2013 09:46:16 +0100 Subject: [Distutils] Merging forks Message-ID: Something prompted by the setuptools merge, and its impact on pip. Does Python's packaging metadata need a way of declaring in a package that it is a merge of (two or more) other packages? Obviously this is a rare scenario, but it's painful when it occurs (how can a tool know that setuptools 0.7 is a later version of distribute than 0.6.36?) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at python.org Tue Jun 11 11:27:02 2013 From: chris at python.org (Chris Withers) Date: Tue, 11 Jun 2013 10:27:02 +0100 Subject: [Distutils] buildout 1 still not managing to pin buildout to 1.x In-Reply-To: References: <51B5A8C9.6050508@simplistix.co.uk> Message-ID: <51B6ED66.8060002@python.org> On 10/06/2013 18:57, Tres Seaver wrote: > Works for me: > > $ mkdir /tmp/withers > $ cd /tmp/withers > $ echo "[buildout] > parts ="> buildout.cfg > $ wget http://downloads.buildout.org/1/bootstrap.py > ... > $ /opt/Python-2.5.x/bin/python bootstrap.py Try with 2.7, and also try going from 1 to 2 and back again by switching bootstraps. There's likely something funky going on (.installed.cfg, develop-eggs or some other goat sacrifice in bin/) but it definitely happened. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From donald at stufft.io Tue Jun 11 12:11:52 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 11 Jun 2013 06:11:52 -0400 Subject: [Distutils] Merging forks In-Reply-To: References: Message-ID: <361A32A0-7DE5-47D4-B547-965313C50ECD@stufft.io> On Jun 11, 2013, at 4:46 AM, Paul Moore wrote: > Something prompted by the setuptools merge, and its impact on pip. Does Python's packaging metadata need a way of declaring in a package that it is a merge of (two or more) other packages? > > Obviously this is a rare scenario, but it's painful when it occurs (how can a tool know that setuptools 0.7 is a later version of distribute than 0.6.36?) > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig http://www.python.org/dev/peps/pep-0426/#obsoleted-by ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From hrsinghal68 at gmail.com Tue Jun 11 14:30:02 2013 From: hrsinghal68 at gmail.com (Harsh Raj Singhal) Date: Tue, 11 Jun 2013 14:30:02 +0200 Subject: [Distutils] Pylons : include git hash inside distribution package Message-ID: Hello, I want to include latest git hash, whenever I build the egg for Pylons application. Is it possible to do that without breaking the other features of setup.py file. Eg : python setup.py bdist_egg (Is it possible to do it within this command?) Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From kencochrane at gmail.com Tue Jun 11 14:31:47 2013 From: kencochrane at gmail.com (ken cochrane) Date: Tue, 11 Jun 2013 08:31:47 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: <5E6CBF13-A2B9-4699-A2B9-3C046F489C25@coderanger.net> References: <39538F60-678B-42CB-B17D-B62B58409C07@stufft.io> <5E6CBF13-A2B9-4699-A2B9-3C046F489C25@coderanger.net> Message-ID: OK, http://www.pypi-mirrors.org is now reporting b.pypi.python.org as N/A which I believe means the DNS propagation is complete. We can move on to phase 2 and use Brett's trick to switch to serving 404's for everything for a little while (how long?) and then fully turn off the app. Ken On Sun, Jun 9, 2013 at 3:46 PM, Noah Kantrowitz wrote: > > On Jun 7, 2013, at 2:34 PM, Noah Kantrowitz wrote: > > > > > On Jun 7, 2013, at 2:27 PM, Donald Stufft wrote: > > > >> > >> On Jun 7, 2013, at 5:26 PM, ken cochrane wrote: > >> > >>> > >>> > >>>> b.pypi.python.org is an official mirror that runs on Google App > engine, and it uses a special mirror package built just for GAE. Code for > it is found here. https://bitbucket.org/loewis/pypi-appengine > >>>> > >>>> b.pypi.python.org has been broken for over 104 days according to > http://www.pypi-mirrors.org, and this is because of an issue when we > switched pypi over to serving over SSL. > >>>> > >>>> I have submitted a pull request to fix this. > https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. > >>>> > >>>> I am one of the maintainers of b.pypi.python.org, so I can see the > logs and push out a new version. > >>>> I haven't needed to push a version out before, and I'm a little > hesitant incase I do it wrong and break something. > >>>> > >>>> I also don't want to push code to GAE from my fork, until my PR gets > accepted or else someone else in the future might deploy the original one > again and remove my fix. > >>>> > >>>> Two things: > >>>> > >>>> 1. Now that we have the pypi CDN up, do we still need this mirror? > >>> > >>> Honestly probably not. Mirrors are less important from a > availability/speed side of things now and will likely move to being more > useful for companies and such to use. > >>> > >>> OK, what would be the procedure for removing a mirror? Anyone know who > is in charge of this mirror? I think Guido had it setup when he worked at > Google, and google is paying for the costs of the mirror, but now that he > doesn't work for Google, not sure who might be the contact person on that > side. > >>> > >>> I've asked Guido who are admins on the account it to get it turned > off. If he doesn't know I can try to find out internally. > >>> > >>> > >>> Brett, > >>> > >>> Here is the owner list that I can see on GAE > >>> > >>> - guido > >>> - kencochrane (me) > >>> - kumar.mcmillan > >>> - martin.v.loewis > >>> - r1chardj0n3s > >>> > >>> So Guido didn't even know he was an owner. =) > >>> > >>> In terms of shutting down the app, you will want to do two things. > First is empty out the cron.yaml file; it should have nothing more than > "cron:". After that you probably want to return 404 for everything; see > http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. > >>> > >>> If you want, Ken, I can clone > https://bitbucket.org/kencochrane/pypi-appengine and send you a pull > request to do all of this since I recently did something similar for > py3ksupport.appspot.com when I shut it down. > >>> > >>> Brett, > >>> > >>> If our goal is to shut it down, then yes please, if you can send a > pull request that would be great.. We should also probably remove it from > the pypi mirror pool before we do this so it no longer gets traffic sent > it's way. > >>> > >>> > >>> > >>> > >>> > >>> If we can get it up to date again, I think it is fine, but an out of > date mirror is not useful to anyone, and it could cause problems in the > long run. > >>> > >>>> > >>>> 2. If yes to 1. if someone can take a minute to review my PR, and > leave comments, or if you have the power, accept my pull request and push > out a new version so we can get the mirror up to date. > >>> > >>> I don't have such permission sadly. > >>> > >>> > >>> Thank you anyway. > >>> > >>> > >>> > >>> > >>> > >> > >> Paging Noah to kill the DNS Unless there's any objections to removing > it? > > > > If no one complains in the next 24-ish hours I'll just point it back at > pypi.python.org like we did with d.pypi. > > This is now complete. In another 24 hours there should be no traffic to > the GAE app and it can just be archived or deleted. > > --Noah > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Wed Jun 12 01:22:11 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 11 Jun 2013 19:22:11 -0400 Subject: [Distutils] buildout 1 still not managing to pin buildout to 1.x In-Reply-To: <51B6ED66.8060002@python.org> References: <51B5A8C9.6050508@simplistix.co.uk> <51B6ED66.8060002@python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/11/2013 05:27 AM, Chris Withers wrote: > On 10/06/2013 18:57, Tres Seaver wrote: >> Works for me: >> >> $ mkdir /tmp/withers $ cd /tmp/withers $ echo "[buildout] parts ="> >> buildout.cfg $ wget http://downloads.buildout.org/1/bootstrap.py >> ... $ /opt/Python-2.5.x/bin/python bootstrap.py > > Try with 2.7, and also try going from 1 to 2 and back again by > switching bootstraps. > > There's likely something funky going on (.installed.cfg, develop-eggs > or some other goat sacrifice in bin/) but it definitely happened. That same recipe works with 2.7. In general, the presence of .installed.cfg should be thought of as problematic for any big surgery (like changing bootstraps, or major versions of buildout), because it hampers repeatability in order to make the build run faster. When in doubt, remove .installed.cfg and re-run buildout after any such change. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG3sSMACgkQ+gerLs4ltQ4ExACfaXdJ1n3jFV+W0HmuNuKB1SDN 8W0Anj3WVTQebSJ71K1Q488KCr5JK/gd =NsB8 -----END PGP SIGNATURE----- From fred at fdrake.net Wed Jun 12 01:48:44 2013 From: fred at fdrake.net (Fred Drake) Date: Tue, 11 Jun 2013 19:48:44 -0400 Subject: [Distutils] buildout 1 still not managing to pin buildout to 1.x In-Reply-To: References: <51B5A8C9.6050508@simplistix.co.uk> <51B6ED66.8060002@python.org> Message-ID: On Tue, Jun 11, 2013 at 7:22 PM, Tres Seaver wrote: > When in doubt, remove .installed.cfg and re-run buildout after any such > change. So the real question is: Why doesn't bootstrap.py just do this? I've stumbled on this many times as well; I don't see an advantage to this not being handled by the bootstrap, because it will surprise us eventually. Some of us (include me, apparently) are more easily surprised than others. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From jim at zope.com Wed Jun 12 03:25:30 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 11 Jun 2013 21:25:30 -0400 Subject: [Distutils] buildout 1 still not managing to pin buildout to 1.x In-Reply-To: References: <51B5A8C9.6050508@simplistix.co.uk> <51B6ED66.8060002@python.org> Message-ID: On Tue, Jun 11, 2013 at 7:22 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/11/2013 05:27 AM, Chris Withers wrote: >> On 10/06/2013 18:57, Tres Seaver wrote: >>> Works for me: >>> >>> $ mkdir /tmp/withers $ cd /tmp/withers $ echo "[buildout] parts ="> >>> buildout.cfg $ wget http://downloads.buildout.org/1/bootstrap.py >>> ... $ /opt/Python-2.5.x/bin/python bootstrap.py >> >> Try with 2.7, and also try going from 1 to 2 and back again by >> switching bootstraps. >> >> There's likely something funky going on (.installed.cfg, develop-eggs >> or some other goat sacrifice in bin/) but it definitely happened. > > That same recipe works with 2.7. In general, the presence of > .installed.cfg should be thought of as problematic for any big surgery > (like changing bootstraps, or major versions of buildout), because it > hampers repeatability in order to make the build run faster. No. It is not an optimization. It's there to know what needs to be uninstalled. It's often OK to just remove it. but by doing so you're forgoing cleanup. It would be better to uninstall everything with the old buildout: bin/buildout buildout:parts= Before removing the installed file. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Wed Jun 12 03:32:08 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 11 Jun 2013 21:32:08 -0400 Subject: [Distutils] buildout 1 still not managing to pin buildout to 1.x In-Reply-To: References: <51B5A8C9.6050508@simplistix.co.uk> <51B6ED66.8060002@python.org> Message-ID: On Tue, Jun 11, 2013 at 7:48 PM, Fred Drake wrote: > On Tue, Jun 11, 2013 at 7:22 PM, Tres Seaver wrote: >> When in doubt, remove .installed.cfg and re-run buildout after any such >> change. > > So the real question is: Why doesn't bootstrap.py just do this? Because it's not just an optimization and removing .installed.cfg prevents cleanup that might be desirable. Removing it as part of bootstrap seems too magic for me. I'd rather: - Error if there's a .installed.cfg - adding a --force option that tries to uninstall and then removed .instaled.cfg. > I've stumbled on this many times as well; I don't see an advantage to > this not being handled by the bootstrap, because it will surprise us > eventually. It rarely surprises me. > Some of us (include me, apparently) are more easily surprised than others. No kidding. ;) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From malte.forkel at berlin.de Wed Jun 12 19:09:39 2013 From: malte.forkel at berlin.de (Malte Forkel) Date: Wed, 12 Jun 2013 19:09:39 +0200 Subject: [Distutils] Binary installer on Windows can't install extension Message-ID: Hello, I have a problem with an install script in Python 2.7.5 on Windows 7 64-Bit. I have written an install script to register a COM server for a Windows-specific package. I specify the script with the --install-script option to the bdist_wininst command. When I run the installer, the install script fails trying to install an extension: Traceback (most recent call last): ... File "C:\Program Files\Python\lib\site-packages\mx\DateTime\DateTime.py", line 9, in from mxDateTime import * File "C:\Program Files\Python\lib\site-packages\mx\DateTime\mxDateTime\__init__.py", line 13, in raise ImportError, why ImportError: DLL load failed: %1 is not a valid Win32 application. close failed in file object destructor: sys.excepthook is missing lost sys.stderr The last three lines of output are probably caused by a known, unrelated issue (http://bugs.python.org/issue15321). I can run the install script by hand without problems. Importing mx.DateTime in Python works fine as well. I know very litte about Windows internals, but have a guess nevertheless: I checked mx\DateTime\mxDateTime\mxDateTime.pyd with Dependency Walker (depends.exe). It seems, the extension does not find the proper version of MSVCR90.DLL. Instead, it is trying to use a 32-bit version of the DLL that was installed as part of a 32-bit application. The application's directory is included in the user search path (PATH). Obviously, a 64-bit version of the DLL exists and is used by Python itself. What is causing my problem? How can I solve it? Thanks, Malte From zygmunt.krynicki at canonical.com Wed Jun 12 22:50:31 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 12 Jun 2013 22:50:31 +0200 Subject: [Distutils] Pylons : include git hash inside distribution package In-Reply-To: References: Message-ID: Yes have a look at versiontools on pypi 11 cze 2013 14:30, "Harsh Raj Singhal" napisa?(a): > Hello, > > I want to include latest git hash, whenever I build the egg for Pylons > application. Is it possible to do that without breaking the other features > of setup.py file. > > Eg : python setup.py bdist_egg (Is it possible to do it within this > command?) > > > Thank you > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaraco at jaraco.com Thu Jun 13 16:57:04 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Thu, 13 Jun 2013 14:57:04 +0000 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: References: <05c820d2d5984850b6dd8543ae3af5f9@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: <2e105b1fa9e64ea0bfbbbf143ac8be02@BLUPR06MB003.namprd06.prod.outlook.com> Hi everyone. I?ve filed Setuptools #18 (https://bitbucket.org/pypa/setuptools/issue/18/packages-that-use_setuptools-from-older) to track this issue. I?ve had some time to think about it this week, and I think I have a workaround that I?ll post on the tracker. Thanks for the reports. From: Julian Berman [mailto:julian at grayvines.com] Sent: Sunday, 09 June, 2013 14:46 To: Jason R. Coombs Cc: distutils-sig at python.org Subject: Re: [Distutils] Setuptools/Distribute error with 0.7.2 Hey Jason. I was getting it before you removed it I think (and yeah, still get it now if I `pip install https://bitbucket.org/pypa/setuptools/downloads/distribute-0.7.zip` ). An example of such a package is pudb, if you'd like to try and reproduce (or I can open an issue on the tracker, just wasn't sure I wasn't missing something), but even if I install with easy_install I still seem to get the same output: ? easy_install pudb Julian at air Searching for pudb Reading https://pypi.python.org/simple/pudb/ Best match: pudb 2013.2 Downloading https://pypi.python.org/packages/source/p/pudb/pudb-2013.2.tar.gz#md5=16411aaffc6f7ae63aba3830ba4ebee5 Processing pudb-2013.2.tar.gz Writing /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/setup.cfg Running pudb-2013.2/setup.py -q bdist_egg --dist-dir /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/egg-dist-tmp-6RBU6H Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35 Building a Distribute egg in /private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2 Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /usr/local/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/distribute-0.6.35-py2.7.egg error: None ~ ? pip freeze | grep distribute Julian at air distribute==0.7 On Sun, Jun 9, 2013 at 2:29 PM, Jason R. Coombs > wrote: Hi Julian, I believe for those packages that call `use_setuptools()` from `distribute_setup.py`, the only option to support them is to install Distribute 0.7 (the stub package), which I believe will satisfy the use_setuptools() call so it doesn?t attempt to install Distribute 0.6. As mentioned in another post, I?ve temporarily pulled Distribute 0.7 from PyPI until we can determine a way to avoid the pip upgrade issue. In the meantime, if you install Distribute 0.7 from https://bitbucket.org/pypa/setuptools/downloads, does that solve your issue? Regards, Jason From: Distutils-SIG [mailto: distutils-sig-bounces+jaraco= jaraco.com at python.org] On Behalf Of Julian Berman Sent: Sunday, 09 June, 2013 14:21 To: distutils-sig at python.org Subject: Re: [Distutils] Setuptools/Distribute error with 0.7.2 (I was also getting this until I used easy_install to upgrade) What is supposed to happen with stuff that used to do `use_setuptools()` in their setup.py now? I upgraded to setuptools 0.7.2, but now if I try to install a thing that has that in the setup.py, it seems to want to try and install distribute, and I get: Downloading/unpacking pudb (from -r /Users/Julian/.local/share/virtualenvs/requirements-every-virtualenv.txt (line 2)) Using download cache from /Users/Julian/Library/Caches/pip/https%3A%2F%2Fpypi.python.org %2Fpackages%2Fsource%2Fp%2Fpudb%2Fpudb-2013.2.tar.gz Running setup.py egg_info for package pudb Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3 Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 Building a Distribute egg in /Users/Julian/.local/share/virtualenvs/great/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg Traceback (most recent call last): File "", line 16, in File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. Complete output from command python setup.py egg_info: Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3 Now working in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 Building a Distribute egg in /Users/Julian/.local/share/virtualenvs/great/build/pudb Traceback (most recent call last): File "setup.py", line 45, in exec(init_file.read(), d) File "", line 8, in File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", line 6, in from setuptools.command.install import install File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", line 8, in from setuptools.command import install_scripts File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", line 3, in from pkg_resources import Distribution, PathMetadata, ensure_directory File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 710, in subscribe callback(dist) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2825, in add_activation_listener(lambda dist: dist.activate()) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2257, in activate self.insert_on(path) File "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", line 2358, in insert_on "with distribute. Found one at %s" % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg Traceback (most recent call last): File "", line 16, in File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, in use_setuptools() File "distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. ---------------------------------------- Command python setup.py egg_info failed with error code 1 in /Users/Julian/.local/share/virtualenvs/great/build/pudb S -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From qwcode at gmail.com Thu Jun 13 18:02:19 2013 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 13 Jun 2013 09:02:19 -0700 Subject: [Distutils] Setuptools/Distribute error with 0.7.2 In-Reply-To: <2e105b1fa9e64ea0bfbbbf143ac8be02@BLUPR06MB003.namprd06.prod.outlook.com> References: <05c820d2d5984850b6dd8543ae3af5f9@BLUPR06MB003.namprd06.prod.outlook.com> <2e105b1fa9e64ea0bfbbbf143ac8be02@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: to prevent any confusion, there's 2 different issues in this thread 1) what Jason just mentioned, the "use_setuptools" issue. https://bitbucket.org/pypa/setuptools/issue/18/packages-that-use_setuptools-from-older 2) using pip to upgrade to the new setuptools: https://github.com/pypa/pip/issues/986 Marcus On Thu, Jun 13, 2013 at 7:57 AM, Jason R. Coombs wrote: > Hi everyone.**** > > ** ** > > I?ve filed Setuptools #18 ( > https://bitbucket.org/pypa/setuptools/issue/18/packages-that-use_setuptools-from-older) > to track this issue.**** > > ** ** > > I?ve had some time to think about it this week, and I think I have a > workaround that I?ll post on the tracker.**** > > ** ** > > Thanks for the reports.**** > > ** ** > > ** ** > > *From:* Julian Berman [mailto:julian at grayvines.com] > *Sent:* Sunday, 09 June, 2013 14:46 > *To:* Jason R. Coombs > *Cc:* distutils-sig at python.org > > *Subject:* Re: [Distutils] Setuptools/Distribute error with 0.7.2**** > > ** ** > > Hey Jason.**** > > ** ** > > I was getting it before you removed it I think (and yeah, still get it now > if I `pip install > https://bitbucket.org/pypa/setuptools/downloads/distribute-0.7.zip`).**** > > ** ** > > An example of such a package is pudb, if you'd like to try and reproduce > (or I can open an issue on the tracker, just wasn't sure I wasn't missing > something), but even if I install with easy_install I still seem to get the > same output:**** > > ** ** > > ? easy_install pudb > > Julian at air**** > > Searching for pudb**** > > Reading https://pypi.python.org/simple/pudb/**** > > Best match: pudb 2013.2**** > > Downloading > https://pypi.python.org/packages/source/p/pudb/pudb-2013.2.tar.gz#md5=16411aaffc6f7ae63aba3830ba4ebee5 > **** > > Processing pudb-2013.2.tar.gz**** > > Writing > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/setup.cfg > **** > > Running pudb-2013.2/setup.py -q bdist_egg --dist-dir > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/egg-dist-tmp-6RBU6H > **** > > Downloading > http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz > **** > > Extracting in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl > **** > > Now working in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35 > **** > > Building a Distribute egg in > /private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2 > **** > > Traceback (most recent call last):**** > > File "setup.py", line 45, in **** > > exec(init_file.read(), d)**** > > File "", line 8, in **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/__init__.py", > line 2, in **** > > from setuptools.extension import Extension, Library**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/extension.py", > line 5, in **** > > from setuptools.dist import _get_unpatched**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/dist.py", > line 6, in **** > > from setuptools.command.install import install**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/command/__init__.py", > line 8, in **** > > from setuptools.command import install_scripts**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/setuptools/command/install_scripts.py", > line 3, in **** > > from pkg_resources import Distribution, PathMetadata, ensure_directory > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > add_activation_listener(lambda dist: dist.activate())**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", > line 710, in subscribe**** > > callback(dist)**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > add_activation_listener(lambda dist: dist.activate())**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", > line 2257, in activate**** > > self.insert_on(path)**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/temp/tmpBW8hGl/distribute-0.6.35/pkg_resources.py", > line 2358, in insert_on**** > > "with distribute. Found one at %s" % str(self.location))**** > > ValueError: A 0.7-series setuptools cannot be installed with distribute. > Found one at > /usr/local/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg**** > > > /private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/easy_install-URSqF1/pudb-2013.2/distribute-0.6.35-py2.7.egg > **** > > error: None**** > > ** ** > > ~**** > > ? pip freeze | grep distribute > > Julian at air**** > > distribute==0.7**** > > ** ** > > On Sun, Jun 9, 2013 at 2:29 PM, Jason R. Coombs wrote: > **** > > Hi Julian,**** > > I believe for those packages that call `use_setuptools()` > from `distribute_setup.py`, the only option to support them is to install > Distribute 0.7 (the stub package), which I believe will satisfy the > use_setuptools() call so it doesn?t attempt to install Distribute 0.6.**** > > **** > > As mentioned in another post, I?ve temporarily pulled > Distribute 0.7 from PyPI until we can determine a way to avoid the pip > upgrade issue. In the meantime, if you install Distribute 0.7 from > https://bitbucket.org/pypa/setuptools/downloads, does that solve your > issue?**** > > **** > > Regards,**** > > Jason**** > > **** > > *From:* Distutils-SIG [mailto:distutils-sig-bounces+jaraco= > jaraco.com at python.org] *On Behalf Of *Julian Berman > *Sent:* Sunday, 09 June, 2013 14:21 > *To:* distutils-sig at python.org > *Subject:* Re: [Distutils] Setuptools/Distribute error with 0.7.2**** > > **** > > (I was also getting this until I used easy_install to upgrade)**** > > **** > > What is supposed to happen with stuff that used to do `use_setuptools()` > in their setup.py now?**** > > **** > > I upgraded to setuptools 0.7.2, but now if I try to install a thing that > has that in the setup.py, it seems to want to try and install distribute, > and I get:**** > > **** > > Downloading/unpacking pudb (from -r > /Users/Julian/.local/share/virtualenvs/requirements-every-virtualenv.txt > (line 2))**** > > Using download cache from /Users/Julian/Library/Caches/pip/https%3A%2F% > 2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpudb%2Fpudb-2013.2.tar.gz**** > > Running setup.py egg_info for package pudb**** > > Downloading > http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz > **** > > Extracting in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3**** > > Now working in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 > **** > > Building a Distribute egg in > /Users/Julian/.local/share/virtualenvs/great/build/pudb**** > > Traceback (most recent call last):**** > > File "setup.py", line 45, in **** > > exec(init_file.read(), d)**** > > File "", line 8, in **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", > line 2, in **** > > from setuptools.extension import Extension, Library**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", > line 5, in **** > > from setuptools.dist import _get_unpatched**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", > line 6, in **** > > from setuptools.command.install import install**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", > line 8, in **** > > from setuptools.command import install_scripts**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", > line 3, in **** > > from pkg_resources import Distribution, PathMetadata, > ensure_directory**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > add_activation_listener(lambda dist: dist.activate())**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 710, in subscribe**** > > callback(dist)**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > add_activation_listener(lambda dist: dist.activate())**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2257, in activate**** > > self.insert_on(path)**** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2358, in insert_on**** > > "with distribute. Found one at %s" % str(self.location))**** > > ValueError: A 0.7-series setuptools cannot be installed with > distribute. Found one at > /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg > **** > > > /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg > **** > > Traceback (most recent call last):**** > > File "", line 16, in **** > > File > "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", line 5, > in **** > > use_setuptools()**** > > File "distribute_setup.py", line 152, in use_setuptools**** > > return _do_download(version, download_base, to_dir, download_delay) > **** > > File "distribute_setup.py", line 132, in _do_download**** > > _build_egg(egg, tarball, to_dir)**** > > File "distribute_setup.py", line 123, in _build_egg**** > > raise IOError('Could not build the egg.')**** > > IOError: Could not build the egg.**** > > Complete output from command python setup.py egg_info:**** > > Downloading > http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz > **** > > **** > > Extracting in /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3** > ** > > **** > > Now working in > /var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35 > **** > > **** > > Building a Distribute egg in > /Users/Julian/.local/share/virtualenvs/great/build/pudb**** > > **** > > Traceback (most recent call last):**** > > **** > > File "setup.py", line 45, in **** > > **** > > exec(init_file.read(), d)**** > > **** > > File "", line 8, in **** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/__init__.py", > line 2, in **** > > **** > > from setuptools.extension import Extension, Library**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/extension.py", > line 5, in **** > > **** > > from setuptools.dist import _get_unpatched**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/dist.py", > line 6, in **** > > **** > > from setuptools.command.install import install**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/__init__.py", > line 8, in **** > > **** > > from setuptools.command import install_scripts**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/setuptools/command/install_scripts.py", > line 3, in **** > > **** > > from pkg_resources import Distribution, PathMetadata, ensure_directory > **** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > **** > > add_activation_listener(lambda dist: dist.activate())**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 710, in subscribe**** > > **** > > callback(dist)**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2825, in **** > > **** > > add_activation_listener(lambda dist: dist.activate())**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2257, in activate**** > > **** > > self.insert_on(path)**** > > **** > > File > "/private/var/folders/zc/bzrwlgxd51594q6lt1yrx5v00000gn/T/tmp8NepZ3/distribute-0.6.35/pkg_resources.py", > line 2358, in insert_on**** > > **** > > "with distribute. Found one at %s" % str(self.location))**** > > **** > > ValueError: A 0.7-series setuptools cannot be installed with distribute. > Found one at > /Users/Julian/.local/share/virtualenvs/great/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg > **** > > **** > > > /Users/Julian/.local/share/virtualenvs/great/build/pudb/distribute-0.6.35-py2.7.egg > **** > > **** > > Traceback (most recent call last):**** > > **** > > File "", line 16, in **** > > **** > > File "/Users/Julian/.local/share/virtualenvs/great/build/pudb/setup.py", > line 5, in **** > > **** > > use_setuptools()**** > > **** > > File "distribute_setup.py", line 152, in use_setuptools**** > > **** > > return _do_download(version, download_base, to_dir, download_delay)*** > * > > **** > > File "distribute_setup.py", line 132, in _do_download**** > > **** > > _build_egg(egg, tarball, to_dir)**** > > **** > > File "distribute_setup.py", line 123, in _build_egg**** > > **** > > raise IOError('Could not build the egg.')**** > > **** > > IOError: Could not build the egg.**** > > **** > > ----------------------------------------**** > > Command python setup.py egg_info failed with error code 1 in > /Users/Julian/.local/share/virtualenvs/great/build/pudb**** > > S**** > > **** > > ** ** > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Fri Jun 14 00:05:06 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 13 Jun 2013 22:05:06 +0000 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: References: Message-ID: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> You will need a 64-bit version of mxDateTime.pyd when used with 64-bit Python. The DLL load fails because a 64-bit process can only load 64-bit DLLs. If you can't get a 64-bit version, it should not be a problem to install and use a 32-bit version of Python on 64-bit Windows (and I generally recommend it - the only thing you gain with 64-bit Python is a higher memory limit). Cheers, Steve -----Original Message----- From: Distutils-SIG [mailto:distutils-sig-bounces+steve.dower=microsoft.com at python.org] On Behalf Of Malte Forkel Sent: Wednesday, June 12, 2013 1010 To: Distutils-Sig at Python.Org Subject: [Distutils] Binary installer on Windows can't install extension Hello, I have a problem with an install script in Python 2.7.5 on Windows 7 64-Bit. I have written an install script to register a COM server for a Windows-specific package. I specify the script with the --install-script option to the bdist_wininst command. When I run the installer, the install script fails trying to install an extension: Traceback (most recent call last): ... File "C:\Program Files\Python\lib\site-packages\mx\DateTime\DateTime.py", line 9, in from mxDateTime import * File "C:\Program Files\Python\lib\site-packages\mx\DateTime\mxDateTime\__init__.py", line 13, in raise ImportError, why ImportError: DLL load failed: %1 is not a valid Win32 application. close failed in file object destructor: sys.excepthook is missing lost sys.stderr The last three lines of output are probably caused by a known, unrelated issue (http://bugs.python.org/issue15321). I can run the install script by hand without problems. Importing mx.DateTime in Python works fine as well. I know very litte about Windows internals, but have a guess nevertheless: I checked mx\DateTime\mxDateTime\mxDateTime.pyd with Dependency Walker (depends.exe). It seems, the extension does not find the proper version of MSVCR90.DLL. Instead, it is trying to use a 32-bit version of the DLL that was installed as part of a 32-bit application. The application's directory is included in the user search path (PATH). Obviously, a 64-bit version of the DLL exists and is used by Python itself. What is causing my problem? How can I solve it? Thanks, Malte _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org http://mail.python.org/mailman/listinfo/distutils-sig From brett at python.org Fri Jun 14 01:44:47 2013 From: brett at python.org (Brett Cannon) Date: Thu, 13 Jun 2013 19:44:47 -0400 Subject: [Distutils] b.pypi.python.org In-Reply-To: References: <39538F60-678B-42CB-B17D-B62B58409C07@stufft.io> <5E6CBF13-A2B9-4699-A2B9-3C046F489C25@coderanger.net> Message-ID: Pull request sent with the changes to always return a 404 and to shut down cron. On Tue, Jun 11, 2013 at 8:31 AM, ken cochrane wrote: > OK, http://www.pypi-mirrors.org is now reporting b.pypi.python.org as N/A > which I believe means the DNS propagation is complete. We can move on to > phase 2 and use Brett's trick to switch to serving 404's for everything for > a little while (how long?) and then fully turn off the app. > > Ken > > > On Sun, Jun 9, 2013 at 3:46 PM, Noah Kantrowitz wrote: > >> >> On Jun 7, 2013, at 2:34 PM, Noah Kantrowitz wrote: >> >> > >> > On Jun 7, 2013, at 2:27 PM, Donald Stufft wrote: >> > >> >> >> >> On Jun 7, 2013, at 5:26 PM, ken cochrane >> wrote: >> >> >> >>> >> >>> >> >>>> b.pypi.python.org is an official mirror that runs on Google App >> engine, and it uses a special mirror package built just for GAE. Code for >> it is found here. https://bitbucket.org/loewis/pypi-appengine >> >>>> >> >>>> b.pypi.python.org has been broken for over 104 days according to >> http://www.pypi-mirrors.org, and this is because of an issue when we >> switched pypi over to serving over SSL. >> >>>> >> >>>> I have submitted a pull request to fix this. >> https://bitbucket.org/loewis/pypi-appengine/pull-request/2/change-pypi-mirror-connection-to-https/diff#comment-262919 but it hasn't been accepted. >> >>>> >> >>>> I am one of the maintainers of b.pypi.python.org, so I can see the >> logs and push out a new version. >> >>>> I haven't needed to push a version out before, and I'm a little >> hesitant incase I do it wrong and break something. >> >>>> >> >>>> I also don't want to push code to GAE from my fork, until my PR gets >> accepted or else someone else in the future might deploy the original one >> again and remove my fix. >> >>>> >> >>>> Two things: >> >>>> >> >>>> 1. Now that we have the pypi CDN up, do we still need this mirror? >> >>> >> >>> Honestly probably not. Mirrors are less important from a >> availability/speed side of things now and will likely move to being more >> useful for companies and such to use. >> >>> >> >>> OK, what would be the procedure for removing a mirror? Anyone know >> who is in charge of this mirror? I think Guido had it setup when he worked >> at Google, and google is paying for the costs of the mirror, but now that >> he doesn't work for Google, not sure who might be the contact person on >> that side. >> >>> >> >>> I've asked Guido who are admins on the account it to get it turned >> off. If he doesn't know I can try to find out internally. >> >>> >> >>> >> >>> Brett, >> >>> >> >>> Here is the owner list that I can see on GAE >> >>> >> >>> - guido >> >>> - kencochrane (me) >> >>> - kumar.mcmillan >> >>> - martin.v.loewis >> >>> - r1chardj0n3s >> >>> >> >>> So Guido didn't even know he was an owner. =) >> >>> >> >>> In terms of shutting down the app, you will want to do two things. >> First is empty out the cron.yaml file; it should have nothing more than >> "cron:". After that you probably want to return 404 for everything; see >> http://stackoverflow.com/a/189935/236574 on how to do that for all URLs. >> >>> >> >>> If you want, Ken, I can clone >> https://bitbucket.org/kencochrane/pypi-appengine and send you a pull >> request to do all of this since I recently did something similar for >> py3ksupport.appspot.com when I shut it down. >> >>> >> >>> Brett, >> >>> >> >>> If our goal is to shut it down, then yes please, if you can send a >> pull request that would be great.. We should also probably remove it from >> the pypi mirror pool before we do this so it no longer gets traffic sent >> it's way. >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> If we can get it up to date again, I think it is fine, but an out of >> date mirror is not useful to anyone, and it could cause problems in the >> long run. >> >>> >> >>>> >> >>>> 2. If yes to 1. if someone can take a minute to review my PR, and >> leave comments, or if you have the power, accept my pull request and push >> out a new version so we can get the mirror up to date. >> >>> >> >>> I don't have such permission sadly. >> >>> >> >>> >> >>> Thank you anyway. >> >>> >> >>> >> >>> >> >>> >> >>> >> >> >> >> Paging Noah to kill the DNS Unless there's any objections to removing >> it? >> > >> > If no one complains in the next 24-ish hours I'll just point it back at >> pypi.python.org like we did with d.pypi. >> >> This is now complete. In another 24 hours there should be no traffic to >> the GAE app and it can just be archived or deleted. >> >> --Noah >> >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Fri Jun 14 03:05:14 2013 From: richard at python.org (Richard Jones) Date: Fri, 14 Jun 2013 11:05:14 +1000 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: On 10 June 2013 06:43, Marcus Smith wrote: > maybe this was covered before, but I'm looking for clarification on how > pip upgrades are to work under PEP439. > > I see 3 relevant bits right now: > > 1) "attempts to import pip machinery. If it can then the pip command > proceeds as normal" > > i.e., once it has a pip installed, it doesn't automatically upgrade on > subsequent imports. > That's the intention. I've modified that paragraph: """ The Python installation includes an executable called "pip3" (see PEP 394 for naming rationale etc.) that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal. Once the bootstrap process is complete the "pip3" command is no longer the bootstrap but rather the full pip command. """ > 2) "A bootstrap is used in the place of a the full pip code so that we > don't have to bundle pip and also the install tool is upgradeable outside > of the regular Python upgrade timeframe and processes" > > to be clear, what is the "install tool" here? pip? or the bootstrap? > I will edit the PEP to replace "install tool" with "pip": """ A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also pip is upgradeable outside of the regular Python upgrade timeframe and processes. """ 3) "Manual installation of the pip implementation will be supported through > the manual download of the wheel file and "pip3 install file>. This installation will not perform standard pip installation steps > of saving the file to a cache directory or updating any local database of > installed files."" > > I guess this means "manual upgrades" as well? > this special logic is to be inside pip itself? or an override of "install" > in the bootstrap (that detects when it's a pip wheel)? > I've modified this section to read: """ Some users may have no Internet access suitable for fetching the pip implementation file. Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip3 install ". This installation - since it uses only the bootstrap code - will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files. """ These changes will be online soon. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 14 03:53:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Jun 2013 11:53:51 +1000 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: On 14 June 2013 11:05, Richard Jones wrote: > On 10 June 2013 06:43, Marcus Smith wrote: >> >> maybe this was covered before, but I'm looking for clarification on how >> pip upgrades are to work under PEP439. >> >> I see 3 relevant bits right now: >> >> 1) "attempts to import pip machinery. If it can then the pip command >> proceeds as normal" >> >> i.e., once it has a pip installed, it doesn't automatically upgrade on >> subsequent imports. > > > That's the intention. I've modified that paragraph: > > """ > The Python installation includes an executable called "pip3" (see PEP 394 > for > naming rationale etc.) that attempts to import pip machinery. If it can > then > the pip command proceeds as normal. If it cannot it will bootstrap pip by > downloading the pip implementation wheel file. Once installed, the pip > command proceeds as normal. Once the bootstrap process is complete the > "pip3" > command is no longer the bootstrap but rather the full pip command. > """ The fact that pip just ships a setuptools entry point wrapper as /usr/bin/pip makes this easier. The difference between the stdlib bootstrap script and the ordinary script is that the bootstrap one will: 1. Import pip directly rather than using the pkg_resources entry point mechanism 2. Fallback to retrieving pip from PyPI if that fails 3. Invoke pip.main directly rather than using the pkg_resources entry point mechanism >> 3) "Manual installation of the pip implementation will be supported >> through the manual download of the wheel file and "pip3 install > wheel file>. This installation will not perform standard pip installation >> steps of saving the file to a cache directory or updating any local database >> of installed files."" >> >> I guess this means "manual upgrades" as well? >> this special logic is to be inside pip itself? or an override of "install" >> in the bootstrap (that detects when it's a pip wheel)? > > > I've modified this section to read: > > """ > Some users may have no Internet access suitable for fetching the pip > implementation file. Manual installation of the pip implementation will be > supported through the manual download of the wheel file and "pip3 install > ". This installation - since it uses only the > bootstrap > code - will not perform standard pip installation steps of saving the file > to > a cache directory or updating any local database of installed files. > """ > > These changes will be online soon. I've come to doubt the wisdom of omitting the metadata or using a default install location that differs from pip's default location. Once we download the pip wheel file, we should be able to unpack it to a secure temporary directory and use it to install itself. That means we should be able to do a full install of pip by default, rather than anything that hides the metadata. Updating it later then becomes the same as updating any other pip installed distribution. The Linux distros can deal with it by either preinstalling pip as part of the python packages, or just leave the bootstrap script out of the Python packages and provide a distinct python-pip package as they do now. "sudo yum install python-pip" and "sudo apt-get install python-pip" are already pretty easy ways to bootstrap pip - it's Windows that really needs the help, and a "it's just like any other pip maintained package" approach is highly desirable there. The only trick would be ensuring the pip wheel console script doesn't collide with the bootstrap script, but worst case, we just special case that directly in pip. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From techtonik at gmail.com Fri Jun 14 15:59:15 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 14 Jun 2013 16:59:15 +0300 Subject: [Distutils] PyPI Download Counts In-Reply-To: References: <20130527072731.GX7789@merlinux.eu> <74512CF2-8B20-442C-ADD8-C35C5B3D2F34@stufft.io> <09E702A9-C03F-4F7E-A894-1C2E8EC55457@stufft.io> <20130610073432.GK7789@merlinux.eu> Message-ID: On Mon, Jun 10, 2013 at 10:51 AM, Nick Coghlan wrote: > On 10 June 2013 17:34, holger krekel wrote: > > On Mon, Jun 10, 2013 at 11:35 +1000, Nick Coghlan wrote: > >> On 10 June 2013 06:04, Alex Clark wrote: > >> > Donald Stufft stufft.io> writes: > >> > > >> >> So yes. I broke Download counts because they were not more important > than > >> >> people being able to actually use PyPI to install from. > >> > > >> > > >> > FWIW: > >> > > >> > > >> > You missed the moral of the story: when you make a decision like this, > >> > someone will *always* disagree with you (even over the most trivial > things). > >> > And even if they don't, they may disagree with your approach (e.g. > why not > >> > sort problems with download counts before enabling the CDN) So the > only way > >> > to make everyone happy is to consider everyone who will be affected > by your > >> > actions, before you take action. > >> > >> And, indeed, we plan to run future changes of this magnitude through > >> the PEP process for exactly this reason. We can't promise not to break > >> some features in order to achieve gains we think are worth the loss, > >> but we *can* promise not to break such features without advance > >> warning and explicit consideration of alternatives that may allow us > >> to avoid the breakage in the first place. > And the necessary precondition for that is status.python.org - heartbeat indicator, development roadmap and a dashboard with current issues that needs response from the community. > In the specific case of PyPI, mainly Donald (as the one doing almost > all the development work for the recent PyPI improvements), Richard > (as the PyPI development lead) and Noah (as the python.org > infrastructure lead). I'm less directly involved in PyPI changes (I'm > still working mostly on the next draft of PEP 426 rather than anything > more near term), but I'm still involved enough to want to say "we" > rather than "they" :) > > The main issue that arose with the PyPI CDN change was that while it > *was* discussed, the discussion happened on infrastructure-sig rather > than here, so most PyPI client developers didn't hear about it until > after the switch was already flipped. > Right. You need to scrape several mailing lists and then explicitly monitor the threads that are interesting to be aware of the stuff that happening. Send thanks to people writing infrastructure reports - they are really helpful. status.python.org can contain such summaries prepared collaboratively for release using Etherpad. In this case you can expect more people to pop in. Also, I must add that the PSF blog and Python Insider blogs are not interesting. Sorry guys, but the stats don't lie. There should be a more technical and concise blog resource. Python community should be more inclusive for people without advanced English, and provide some tasty for them too. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Jun 14 16:05:41 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 14 Jun 2013 17:05:41 +0300 Subject: [Distutils] Fixing PyPI download stats with real-time log analysis (Was: PyPI Download Counts) Message-ID: On Sun, Jun 9, 2013 at 9:33 PM, Donald Stufft wrote: > > Fastly has been a dream to work with. They've been fast at fixing and > diagnosing issues, have helped tune the config to get a higher hit rate, > and when they heard that people were upset that download counts > had to be turned off they offered the logging support to be turned on > for our account. > > The infrastructure is setup to receive the logs (and intact was receiving > logs for a day or so) but upgrades to the VM that runs PyPI needs to > occur before we can continue receiving them. The disk drive that PyPI > has is to small to handle the volume of request data that is coming in > from Fastly and it quickly filled up in under 24 hours. Upgrading that > space requires powering off the VM so we (The Infra team) are working > on doing that, ideally without downtime on PyPI. > Disk drives? It is unneeded bottleneck. What is needed is resident log cruncher that processes log entries storing intermediate results in memcache, and updates DB counts in batch. This can be even done on AppEngine. Something makes me believe that we can get some extra free quota for this purpose. =) Could you, please, share the log format+example, so that we can experiment with it? -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Fri Jun 14 16:47:29 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 14 Jun 2013 14:47:29 +0000 Subject: [Distutils] devpi-0.9.1: pushing tested releases to pypi Message-ID: <20130614144729.GW7789@merlinux.eu> Hi all, devpi-0.9.1 is out which fixes bugs and introduces support for pushing a tested release candidate from a private index to pypi. See http://doc.devpi.net on the ease of doing "devpi upload", "test" and "push" commands as well as general information on the devpi-server and devpi tools. Thanks to all issue reporters and to Andi Albrecht for his pull request. best, holger From malte.forkel at berlin.de Fri Jun 14 17:46:22 2013 From: malte.forkel at berlin.de (Malte Forkel) Date: Fri, 14 Jun 2013 17:46:22 +0200 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> References: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: <51BB3ACE.20702@berlin.de> Am 14.06.2013 00:05, schrieb Steve Dower: > You will need a 64-bit version of mxDateTime.pyd when used with 64-bit Python. > The DLL load fails because a 64-bit process can only load 64-bit DLLs. > > If you can't get a 64-bit version, it should not be a problem to install and use > a 32-bit version of Python on 64-bit Windows (and I generally > recommend it - > the only thing you gain with 64-bit Python is a higher memory limit). > I'm afraid its not that easy. Everything I installed is 64-bit, both Python itself and the eGenix mx Base Distribution which includes mx.DateTime. According to Dependency Walker, MSVCR90.DLL is the only 32-bit DLL referenced by mxDateTime.pyd. All others DLLs (inluding the version of MSVCR90.DLL referenced by PYTHON27.DLL) are 64-bit. When imported from a regular Python process, mxDateTime loads fine. The problem only occurred when mxDateTime was imported in the context of the installer's (post)install script. Note the past tense, though. When I ran my tests today, there was no error anymore. I guess if I knew what change has caused that, I knew what caused the initial problem. I started my tests with Python 2.7.3 and mx 2.3.1, then upgraded to 2.7.5 and 2.3.6 respectively. The problem still occurred. Since, I have rebooted the computer and installed another Python package of my own. The paths (user, system) have not changed. From qwcode at gmail.com Sat Jun 15 01:58:01 2013 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 14 Jun 2013 16:58:01 -0700 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: > I've come to doubt the wisdom of omitting the metadata or using a > default install location that differs from pip's default location. > Once we download the pip wheel file, we should be able to unpack it to > a secure temporary directory and use it to install itself. That means > we should be able to do a full install of pip by default, rather than > anything that hides the metadata. Updating it later then becomes the > same as updating any other pip installed distribution. > ok, sounds good. once the pip bootstrap is done, you'll just have a standard install of pip. and after that, it's pip's job to support upgrading itself properly for users. > The Linux distros can deal with it by either preinstalling pip as part > of the python packages, or just leave the bootstrap script out of the > Python packages and provide a distinct python-pip package as they do > now. "sudo yum install python-pip" and "sudo apt-get install > python-pip" are already pretty easy ways to bootstrap pip - it's > Windows that really needs the help, and a "it's just like any other > pip maintained package" approach is highly desirable there. > > The only trick would be ensuring the pip wheel console script doesn't > collide with the bootstrap script, but worst case, we just special > case that directly in pip. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 15 02:50:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jun 2013 10:50:41 +1000 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: On 15 June 2013 09:58, Marcus Smith wrote: > >> I've come to doubt the wisdom of omitting the metadata or using a >> default install location that differs from pip's default location. >> Once we download the pip wheel file, we should be able to unpack it to >> a secure temporary directory and use it to install itself. That means >> we should be able to do a full install of pip by default, rather than >> anything that hides the metadata. Updating it later then becomes the >> same as updating any other pip installed distribution. > > > ok, sounds good. > > once the pip bootstrap is done, you'll just have a standard install of pip. > and after that, it's pip's job to support upgrading itself properly for > users. Yep. There are a couple of wrinkles from a user experience/sanity perspective point of view though. 1. We should write the bootstrapped pip to the same location that we're installing the package to (so system global, user local or current venv). 2. We should check we can actually write to that location before downloading the pip wheel. If we can't write to that destination, we should emit the same error message pip itself would in that situation. 3. If we can write to it, we download the wheel, add it directly to the front of sys.path (as Paul suggested), import pip.main from there and then use that to install pip itself before continuing with the rest of the installation (still using the copy from the wheel for that initial invocation). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From qwcode at gmail.com Sat Jun 15 08:02:33 2013 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 14 Jun 2013 23:02:33 -0700 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: > 1. We should write the bootstrapped pip to the same location that > we're installing the package to (so system global, user local or > current venv). > this makes sense for global and user installs, but not venv? when you're in a venv, you're using the pip installed in that specific venv, not the bootstrap. and just to confirm about user install bootstrapping, it would work like this? - user1 happens to run "pip3 install --user django" as his first pip3 command - the bootstrap installs pip to /home/user1/.local - thereafter, user1's executions of pip3 detects the importable pip in user local, and does no more bootstrapping - user2 runs "pip3 install --user django" as his first pip3 command - the bootstrap installs pip to /home/user2/.local ... and if instead user1 ran "sudo pip install django" as his first command, pip get's installed globally. and thereafter, all users would be using a global installed pip, no matter what they're first command was. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 15 09:25:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jun 2013 17:25:34 +1000 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: On 15 June 2013 16:02, Marcus Smith wrote: > >> 1. We should write the bootstrapped pip to the same location that >> we're installing the package to (so system global, user local or >> current venv). > > > this makes sense for global and user installs, but not venv? > when you're in a venv, you're using the pip installed in that specific venv, > not the bootstrap. True for virtualenv, but not for pyvenv - that doesn't install anything into the virtual environment, it just creates it. I expect the 3.4 version will add a link to the appropriate bootstrapping script though, so we need to define the expected behaviour in that case. > and just to confirm about user install bootstrapping, it would work like > this? > - user1 happens to run "pip3 install --user django" as his first pip3 > command > - the bootstrap installs pip to /home/user1/.local > - thereafter, user1's executions of pip3 detects the importable pip in user > local, and does no more bootstrapping > - user2 runs "pip3 install --user django" as his first pip3 command > - the bootstrap installs pip to /home/user2/.local > ... > > and if instead user1 ran "sudo pip install django" as his first command, pip > get's installed globally. > and thereafter, all users would be using a global installed pip, no matter > what they're first command was. That is what I was saying, but with it spelled out like that, I don't like it. The destination of the current install request isn't adequate justification for defining the location of the installed pip. Instead, I'm starting to think that with the right error message, "Explicit is better than implicit" and "In the face of ambiguity, refuse the temptation to guess" should rule the day here. For example: $pip3 install Django pip is not currently installed. Run 'pip3 bootstrap' or 'pip3 bootstrap --system' to install it. $pip3 bootstrap --help Usage: pip3 bootstrap pip3 bootstrap --system Description: Downloads and installs the latest version of pip from the Python Package Index (pypi.python.org). Installs into the active virtual environment (if any) or the current user's site-packages directory. Has no effect if pip is already installed (run "pip install --upgrade pip" to upgrade to a later version). Options: --system Install into the system site-packages instead of the venv or user site-packages. --wheel Installs from the given wheel file instead of downloading from PyPI Linux distros could then patch the 'pip3 bootstrap --system' variant to invoke the appropriate platform specific installation command. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 15 09:27:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jun 2013 17:27:09 +1000 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: On 15 June 2013 17:25, Nick Coghlan wrote: > $pip3 bootstrap --help > Usage: > pip3 bootstrap > pip3 bootstrap --system Oops, that part would just be: Usage: pip3 bootstrap [options] Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From malte.forkel at berlin.de Sat Jun 15 11:24:11 2013 From: malte.forkel at berlin.de (Malte Forkel) Date: Sat, 15 Jun 2013 11:24:11 +0200 Subject: [Distutils] bdist_msi install-script observations Message-ID: Hello, two things I've noticed when testing an install script with bdist_msi: - The installer requests admin privileges for an installation for all users (on Windows 7), but the install script does not seem to be executed with admin privileges. E.g., execution of win32.com.server.register.RegisterClasses fails with "pywintypes.error: (5, 'RegSetValue', 'Access denied')" - If an error occurrs in the install script, the installer claims "Your system has not been changed" when in fact all files have been installed. My fault, bug, or feature? Thanks, Malte From ethan at stoneleaf.us Sat Jun 15 08:29:55 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 14 Jun 2013 23:29:55 -0700 Subject: [Distutils] error using easy_install Message-ID: <51BC09E3.3040601@stoneleaf.us> My setup.py: -- 8< -------------------------------------------------------------- from distutils.core import setup long_desc = open('enum.rst').read() setup( name='stoneleaf.enum', version='1.0.1', url='https://pypi.python.org/pypi/stoneleaf.enum', packages=['enum'], package_dir={'enum':''}, package_data={'enum':['enum.rst', 'enum.pdf', 'test/test_enum.py', 'test/py2_test_enum.py', 'test/py3_test_enum.py']}, license='BSD License', description='Python 3.4 Enum backported to 3.3, 3.2, 3.1, 2.7, 2.6, 2.5, and 2.4', long_description=long_desc, provides=['enum'], author='Ethan Furman', author_email='ethan at stoneleaf.us', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'License :: OSI Approved :: BSD License', 'Programming Language :: Python', 'Topic :: Software Development' ], ) -- 8< -------------------------------------------------------------- Error trying to install it with easy_install after uploading it to PyPI: ethan at hydra:~$ sudo easy_install stoneleaf.enum Searching for stoneleaf.enum Reading http://pypi.python.org/simple/stoneleaf.enum/ Best match: stoneleaf.enum 1.0.1 Downloading http://pypi.python.org/packages/source/s/stoneleaf.enum/stoneleaf.enum-1.0.1.zip#md5=c41abd7ad04f11e8b545cfca7758a4f2 Processing stoneleaf.enum-1.0.1.zip Writing /tmp/easy_install-LY_zzX/stoneleaf.enum-1.0.1/setup.cfg Running stoneleaf.enum-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-LY_zzX/stoneleaf.enum-1.0.1/egg-dist-tmp-IpvhLD error: Setup script exited with error: can't copy 'num.rst': doesn't exist or not a regular file There is no 'num.rst' file in the package -- it should be 'enum.rst'. Any ideas? Is this the right place to post? -- ~Ethan~ From vinay_sajip at yahoo.co.uk Sat Jun 15 18:35:06 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 15 Jun 2013 16:35:06 +0000 (UTC) Subject: [Distutils] Post-merge setuptools Python 3 single-codebase port available Message-ID: Following the setuptools/distribute merge, I've updated my single codebase port of the merged setuptools at [1]. Source of my repo is available at [2] (single-codebase branch). Travis integration has been set up; results are at [3]. Currently, all tests are passing for 2.5, 2.6, 2.7, 3.2, and 3.3. For installation into PEP 405 venvs, a wheel has been set up at [4]. Feedback welcome. Regards, Vinay Sajip [1] https://bitbucket.org/pypa/setuptools/ [2] https://bitbucket.org/vinay.sajip/setuptools/ [3] https://travis-ci.org/vsajip/setuptools/ [4] https://bitbucket.org/vinay.sajip/setuptools/downloads/ From tseaver at palladion.com Sat Jun 15 19:13:57 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 15 Jun 2013 13:13:57 -0400 Subject: [Distutils] error using easy_install In-Reply-To: <51BC09E3.3040601@stoneleaf.us> References: <51BC09E3.3040601@stoneleaf.us> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/15/2013 02:29 AM, Ethan Furman wrote: > My setup.py: > > -- 8< -------------------------------------------------------------- > from distutils.core import setup > > long_desc = open('enum.rst').read() > > setup( name='stoneleaf.enum', version='1.0.1', > url='https://pypi.python.org/pypi/stoneleaf.enum', packages=['enum'], > package_dir={'enum':''}, package_data={'enum':['enum.rst', 'enum.pdf', > 'test/test_enum.py', 'test/py2_test_enum.py', > 'test/py3_test_enum.py']}, license='BSD License', description='Python > 3.4 Enum backported to 3.3, 3.2, 3.1, 2.7, 2.6, 2.5, and 2.4', > long_description=long_desc, provides=['enum'], author='Ethan Furman', > author_email='ethan at stoneleaf.us', classifiers=[ 'Development Status > :: 5 - Production/Stable', 'Intended Audience :: Developers', 'License > :: OSI Approved :: BSD License', 'Programming Language :: Python', > 'Topic :: Software Development' ], ) -- 8< > -------------------------------------------------------------- > > Error trying to install it with easy_install after uploading it to > PyPI: > > ethan at hydra:~$ sudo easy_install stoneleaf.enum Searching for > stoneleaf.enum Reading http://pypi.python.org/simple/stoneleaf.enum/ > Best match: stoneleaf.enum 1.0.1 Downloading > http://pypi.python.org/packages/source/s/stoneleaf.enum/stoneleaf.enum-1.0.1.zip#md5=c41abd7ad04f11e8b545cfca7758a4f2 > > Processing stoneleaf.enum-1.0.1.zip > Writing /tmp/easy_install-LY_zzX/stoneleaf.enum-1.0.1/setup.cfg > Running stoneleaf.enum-1.0.1/setup.py -q bdist_egg --dist-dir > /tmp/easy_install-LY_zzX/stoneleaf.enum-1.0.1/egg-dist-tmp-IpvhLD > error: Setup script exited with error: can't copy 'num.rst': doesn't > exist or not a regular file > > > There is no 'num.rst' file in the package -- it should be 'enum.rst'. > > Any ideas? Is this the right place to post? This is the right place. Your 'package_data' declaration is getting munged:: > /tmp/stone/stoneleaf.enum-1.0.1/build/bdist.linux-i686/egg/setuptools /command/build_py.py(80)build_package_data() -> self.copy_file(os.path.join(src_dir, filename), target) (Pdb) l 75 lastdir = None 76 for package, src_dir, build_dir, filenames in self.data_files: 77 for filename in filenames: 78 target = os.path.join(build_dir, filename) 79 self.mkpath(os.path.dirname(target)) 80 -> self.copy_file(os.path.join(src_dir, filename), target) 81 82 83 def analyze_manifest(self): 84 self.manifest_files = mf = {} 85 if not self.distribution.include_package_data: (Pdb) pp self.data_files [('enum', '', 'build/lib/enum', ['num.rst', 'num.pdf', 'est/test_enum.py', 'est/py2_test_enum.py', 'est/py3_test_enum.py'])] Looking at the 'build_py' step, this line is problematic: package_dir={'enum':''}, When I change it to: package_dir={'enum':'.'}, then the install succeeds. I would probably just make an 'enum' subdir, move the files into it, and be done with it ('setup.py' isn't logically part of the package: it is in the wrapper). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG8oNUACgkQ+gerLs4ltQ4axQCgh1YAyvqC/ZRJs1V6U4jQ66as 9VQAnRmJqhFINjzjCx2eftJfALKj/vsc =o/a+ -----END PGP SIGNATURE----- From tseaver at palladion.com Sat Jun 15 19:15:36 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 15 Jun 2013 13:15:36 -0400 Subject: [Distutils] Post-merge setuptools Python 3 single-codebase port available In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/15/2013 12:35 PM, Vinay Sajip wrote: > Following the setuptools/distribute merge, I've updated my single > codebase port of the merged setuptools at [1]. Source of my repo is > available at [2] (single-codebase branch). Travis integration has been > set up; results are at [3]. > > Currently, all tests are passing for 2.5, 2.6, 2.7, 3.2, and 3.3. > > For installation into PEP 405 venvs, a wheel has been set up at [4]. > > Feedback welcome. Thank you for your work! +100 for an immediate merge. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG8oTgACgkQ+gerLs4ltQ5/FwCgiYZt/q+t6t2emibspuUAYDLG r4UAn07mxKKQWvpNhG1aLgLZsZ2i2WIO =/7Yf -----END PGP SIGNATURE----- From setuptools at bugs.python.org Sun Jun 16 09:45:23 2013 From: setuptools at bugs.python.org (jerry) Date: Sun, 16 Jun 2013 07:45:23 +0000 Subject: [Distutils] [issue151] Jerry Regal Message-ID: <1371368721.12753.YahooMailNeo@web140401.mail.bf1.yahoo.com> New submission from jerry: How are you? Jerry Regal ---------- files: unnamed messages: 723 nosy: jerry status: unread title: Jerry Regal Added file: http://bugs.python.org/setuptools/file91/unnamed _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Jun 16 09:02:31 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 16 Jun 2013 00:02:31 -0700 Subject: [Distutils] error using easy_install In-Reply-To: References: <51BC09E3.3040601@stoneleaf.us> Message-ID: <51BD6307.3000101@stoneleaf.us> On 06/15/2013 10:13 AM, Tres Seaver wrote: > On 06/15/2013 02:29 AM, Ethan Furman wrote: >> >> Any ideas? Is this the right place to post? > > This is the right place. Your 'package_data' declaration is getting munged:: [snip] > Looking at the 'build_py' step, this line is problematic: > > package_dir={'enum':''}, > > When I change it to: > > package_dir={'enum':'.'}, > > then the install succeeds. I would probably just make an 'enum' subdir, > move the files into it, and be done with it ('setup.py' isn't logically > part of the package: it is in the wrapper). I went with the latter solution, which worked fine. Thanks! -- ~Ethan~ From ethan at stoneleaf.us Sun Jun 16 09:13:41 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 16 Jun 2013 00:13:41 -0700 Subject: [Distutils] complicated setup Message-ID: <51BD65A5.3040500@stoneleaf.us> Here's my file layout: / |- setup.py | |- enum / |- __init__.py | |- py2_enum.py | |- py3_enum.py | |- test / |- test_enum.py | |- py2_test_enum.py | |- py3_test_enum.py __init__ and test_enum are both smart enough to pull in the correct code when imported. The issue I am having is this: --8<-------------------------------------------------------------- ethan at hydra:~$ sudo easy_install enum34 [sudo] password for ethan: Searching for enum34 Reading http://pypi.python.org/simple/enum34/ Best match: enum34 0.9 Downloading http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8 Processing enum34-0.9.zip Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg Running enum34-0.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5 SyntaxError: ('invalid syntax', ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, ' def __call__(cls, value, names=None, *, module=None, type=None):\n')) SyntaxError: ('invalid syntax', ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, ' class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) zip_safe flag not set; analyzing archive contents... SyntaxError: ('invalid syntax', ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py', 211, 43, ' def __call__(cls, value, names=None, *, module=None, type=None):\n')) SyntaxError: ('invalid syntax', ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py', 630, 47, ' class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) Adding enum34 0.9 to easy-install.pth file Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg Processing dependencies for enum34 Finished processing dependencies for enum34 --8<-------------------------------------------------------------- distutils is trying to load the py3 versions, which of course fails on a py2 install. The package installs successfully anyway, but if I were a user I would be wondering if the install was trustworthy. It seems to me that I need to either have distutils only install the version appropriate files, or to not try to scan the version inappropriate files, but at this point I do not know how to do either. Any pointers would be greatly appreciated. -- ~Ethan~ From jim at zope.com Mon Jun 17 14:44:30 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 17 Jun 2013 08:44:30 -0400 Subject: [Distutils] Buildout 2.2.0b1 is available fort testing Message-ID: Changes: - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. - Handle both addition and subtraction of elements (+= and -=) on the same key in the same section. Forward-ported from buildout 1.6. - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` error message being emitted by distribute / setuptools. - Extend distutils script generation to support module docstrings and __future__ imports. - Refactored picked versions logic to make it easier to use for plugins. - Use ``get_win_launcher`` API to find Windows launcher (falling back to ``resource_string`` for ``cli.exe``). - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` in current directory during installation (merged from 1.x branch). To try it out, copy the new bootstrap file from: http://downloads.buildout.org/2.2.0b1/bootstrap.py Then run the bootstrap, making sure to include the -t option: python bootstrap.py -t The -t option is necessary to use the non-final buildout release. We'd like to release 2.2 final soon, so your testing is really helpful. Special thanks to Tres Seaver for updating buildout to work with setuptools 0.7 and for reviewing and merging a number of outstanding pull requests. Of course, thanks to the folks to contributed the many other changes in this release. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From p.f.moore at gmail.com Tue Jun 18 14:26:11 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 18 Jun 2013 13:26:11 +0100 Subject: [Distutils] 503 errors from PyPI Message-ID: Whenever I do a PyPI search at the moment, I get Error 503 backend read error backend read error Guru Meditation: XID: 477806858 ________________________________ Varnish cache server Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jun 18 14:27:07 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 18 Jun 2013 08:27:07 -0400 Subject: [Distutils] 503 errors from PyPI In-Reply-To: References: Message-ID: On Jun 18, 2013, at 8:26 AM, Paul Moore wrote: > Whenever I do a PyPI search at the moment, I get > > Error 503 backend read error > backend read error > Guru Meditation: > XID: 477806858 > ________________________________ > Varnish cache server > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Any search, or particular ones? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue Jun 18 14:29:36 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 18 Jun 2013 13:29:36 +0100 Subject: [Distutils] 503 errors from PyPI In-Reply-To: References: Message-ID: On 18 June 2013 13:27, Donald Stufft wrote: > Any search, or particular ones? Seems to be any. I tried "process" and "django" just now. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jun 18 14:31:06 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 18 Jun 2013 08:31:06 -0400 Subject: [Distutils] 503 errors from PyPI In-Reply-To: References: Message-ID: On Jun 18, 2013, at 8:29 AM, Paul Moore wrote: > On 18 June 2013 13:27, Donald Stufft wrote: > Any search, or particular ones? > > Seems to be any. I tried "process" and "django" just now. > Paul Odd both of those work fine for me, investigating. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue Jun 18 14:38:36 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 18 Jun 2013 13:38:36 +0100 Subject: [Distutils] 503 errors from PyPI In-Reply-To: References: Message-ID: On 18 June 2013 13:31, Donald Stufft wrote: > Odd both of those work fine for me, investigating. It's working now - must have been an intermittent thing. Sorry for the noise. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jun 18 14:40:12 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 18 Jun 2013 08:40:12 -0400 Subject: [Distutils] 503 errors from PyPI In-Reply-To: References: Message-ID: <6C403E28-E543-4ABE-9B83-43446B44EAF1@stufft.io> On Jun 18, 2013, at 8:38 AM, Paul Moore wrote: > On 18 June 2013 13:31, Donald Stufft wrote: > Odd both of those work fine for me, investigating. > > It's working now - must have been an intermittent thing. Sorry for the noise. > Paul Ok! For what it's worth. From my searching the "backend read error" just means the connection between the Fastly CDN server and PyPI ended erroneously. So it was probably just some network blips. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Wed Jun 19 00:19:25 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 19 Jun 2013 00:19:25 +0200 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: <51BB3ACE.20702@berlin.de> References: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> <51BB3ACE.20702@berlin.de> Message-ID: <51C0DCED.3000100@egenix.com> Malte Forkel wrote: > Am 14.06.2013 00:05, schrieb Steve Dower: >> You will need a 64-bit version of mxDateTime.pyd when used with 64-bit Python. >> The DLL load fails because a 64-bit process can only load 64-bit DLLs. >> >> If you can't get a 64-bit version, it should not be a problem to install and use >> a 32-bit version of Python on 64-bit Windows (and I generally >> recommend it - > the only thing you gain with 64-bit Python is a > higher memory limit). >> > > I'm afraid its not that easy. Everything I installed is 64-bit, both > Python itself and the eGenix mx Base Distribution which includes > mx.DateTime. > > According to Dependency Walker, MSVCR90.DLL is the only 32-bit DLL > referenced by mxDateTime.pyd. All others DLLs (inluding the version of > MSVCR90.DLL referenced by PYTHON27.DLL) are 64-bit. Are you using our prebuilt or egg distribution of eGenix mx Base, or compiling it from source ? > When imported from a regular Python process, mxDateTime loads fine. The > problem only occurred when mxDateTime was imported in the context of the > installer's (post)install script. > > Note the past tense, though. When I ran my tests today, there was no > error anymore. I guess if I knew what change has caused that, I knew > what caused the initial problem. I started my tests with Python 2.7.3 > and mx 2.3.1, then upgraded to 2.7.5 and 2.3.6 respectively. The problem > still occurred. Since, I have rebooted the computer and installed > another Python package of my own. The paths (user, system) have not changed. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From qwcode at gmail.com Wed Jun 19 01:02:35 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 18 Jun 2013 16:02:35 -0700 Subject: [Distutils] PEP439 and upgrading pip In-Reply-To: References: Message-ID: Instead, I'm starting to think that with the right error message, > "Explicit is better than implicit" and "In the face of ambiguity, > refuse the temptation to guess" should rule the day here. > > For example: > > $pip3 install Django > pip is not currently installed. Run 'pip3 bootstrap' or 'pip3 > bootstrap --system' to install it. > $pip3 bootstrap --help > Usage: > pip3 bootstrap > pip3 bootstrap --system > > sounds reasonable to me. > Linux distros could then patch the 'pip3 bootstrap --system' variant > to invoke the appropriate platform specific installation command. well, e.g. debian patches distutils.command.install with it's /usr/local layer and dist-packages patches. I imagine, those kind of patches would also occur in distlib as well now? (which will be handling the bootstrap installs) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From malte.forkel at berlin.de Wed Jun 19 10:01:46 2013 From: malte.forkel at berlin.de (Malte Forkel) Date: Wed, 19 Jun 2013 10:01:46 +0200 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: <51C0DCED.3000100@egenix.com> References: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> <51BB3ACE.20702@berlin.de> <51C0DCED.3000100@egenix.com> Message-ID: Am 19.06.2013 00:19, schrieb M.-A. Lemburg: > > Are you using our prebuilt or egg distribution of eGenix mx Base, > or compiling it from source ? > I'm using the prebuilt distribution (egenix-mx-base-3.2.6.win-amd64-py2.7.msi). Thanks to the kind advice of Steve Dower, I have since updated OpenVPN, the application had installed the 32-bit version of MSVCR90.DLL in its own directory, to a newer 64-bit version which does not install local CRTs. According to Dependency Walker, it had been OpenVPN's copy of MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the OpenVPN's entry in the user's search path. Now, according to Dependency Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved. But there are no problems when I run the install script or import mx.DateTime myself. From mal at egenix.com Wed Jun 19 12:51:48 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 19 Jun 2013 12:51:48 +0200 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: References: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> <51BB3ACE.20702@berlin.de> <51C0DCED.3000100@egenix.com> Message-ID: <51C18D44.3060209@egenix.com> On 19.06.2013 10:01, Malte Forkel wrote: > Am 19.06.2013 00:19, schrieb M.-A. Lemburg: >> >> Are you using our prebuilt or egg distribution of eGenix mx Base, >> or compiling it from source ? >> > > I'm using the prebuilt distribution > (egenix-mx-base-3.2.6.win-amd64-py2.7.msi). > > Thanks to the kind advice of Steve Dower, I have since updated OpenVPN, > the application had installed the 32-bit version of MSVCR90.DLL in its > own directory, to a newer 64-bit version which does not install local CRTs. > > According to Dependency Walker, it had been OpenVPN's copy of > MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the > OpenVPN's entry in the user's search path. Interesting. I just tried that on my system and indeed, see the reference to OpenVPN as well. The same happens with all other .pyd files in the Python installation. > Now, according to Dependency > Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved. > But there are no problems when I run the install script or import > mx.DateTime myself. We'll investigate that some more. It may have something to do with the way distutils handles manifests for Python extensions at compile/link time. The extensions should get all the MSVCR90.DLL symbols from the DLL loaded into the PYTHON27.DLL process. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-06-18: Released mxODBC Django DE 1.2.0 ... http://egenix.com/go47 2013-07-01: EuroPython 2013, Florence, Italy ... 12 days to go 2013-07-16: Python Meeting Duesseldorf ... 27 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at python.org Wed Jun 19 13:57:53 2013 From: mal at python.org (M.-A. Lemburg) Date: Wed, 19 Jun 2013 13:57:53 +0200 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: <51C18D44.3060209@egenix.com> References: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> <51BB3ACE.20702@berlin.de> <51C0DCED.3000100@egenix.com> <51C18D44.3060209@egenix.com> Message-ID: <51C19CC1.50600@python.org> On 19.06.2013 12:51, M.-A. Lemburg wrote: > On 19.06.2013 10:01, Malte Forkel wrote: >> Am 19.06.2013 00:19, schrieb M.-A. Lemburg: >>> >>> Are you using our prebuilt or egg distribution of eGenix mx Base, >>> or compiling it from source ? >>> >> >> I'm using the prebuilt distribution >> (egenix-mx-base-3.2.6.win-amd64-py2.7.msi). >> >> Thanks to the kind advice of Steve Dower, I have since updated OpenVPN, >> the application had installed the 32-bit version of MSVCR90.DLL in its >> own directory, to a newer 64-bit version which does not install local CRTs. >> >> According to Dependency Walker, it had been OpenVPN's copy of >> MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the >> OpenVPN's entry in the user's search path. > > Interesting. I just tried that on my system and indeed, see the > reference to OpenVPN as well. The same happens with all other .pyd > files in the Python installation. > >> Now, according to Dependency >> Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved. >> But there are no problems when I run the install script or import >> mx.DateTime myself. > > We'll investigate that some more. It may have something to > do with the way distutils handles manifests for Python extensions > at compile/link time. The extensions should get all the MSVCR90.DLL > symbols from the DLL loaded into the PYTHON27.DLL process. The unresolved reference to MSVCR90.DLL is shown for all distutils compiled extensions, so this is most likely the result of distutils removing the manifest from compiled C extensions to force the linker to use the Python DLL's CRT libs - this in return avoids problem with having to place the CRT manifests into each extension directory. I have no idea why the Windows linker attempts to load a x86 DLL into a x64 process. The manifest included with OpenVPN clearly shows that the DLL is for a different platform, even though it uses the same version as the one listed in the python.exe manifest. -- Marc-Andre Lemburg Director Python Software Foundation http://www.python.org/psf/ From malte.forkel at berlin.de Wed Jun 19 16:08:44 2013 From: malte.forkel at berlin.de (Malte Forkel) Date: Wed, 19 Jun 2013 16:08:44 +0200 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: <51C19CC1.50600@python.org> References: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> <51BB3ACE.20702@berlin.de> <51C0DCED.3000100@egenix.com> <51C18D44.3060209@egenix.com> <51C19CC1.50600@python.org> Message-ID: <51C1BB6C.9070206@berlin.de> Am 19.06.2013 13:57, schrieb M.-A. Lemburg: > > The unresolved reference to MSVCR90.DLL is shown for all distutils > compiled extensions, so this is most likely the result of distutils > removing the manifest from compiled C extensions to force the linker > to use the Python DLL's CRT libs - this in return avoids problem with > having to place the CRT manifests into each extension directory. > After my initial problems, this now to works for me. I still don't understand what went wrong initially, though. > I have no idea why the Windows linker attempts to load a x86 DLL into > a x64 process. The manifest included with OpenVPN clearly shows that > the DLL is for a different platform, even though it uses the same > version as the one listed in the python.exe manifest. > I can a confirm this for the manifest included with OpenVPN 2.2.1 which I had installed before: When Dependency Walker showed a reference from mxDateTime.pyd to the 32-bit version of MSVCR90.DLL, it always also showed a reference from PYTHON27.DLL to the 64-bit version of MSVCR90.DLL. From Steve.Dower at microsoft.com Wed Jun 19 18:30:16 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 19 Jun 2013 16:30:16 +0000 Subject: [Distutils] Binary installer on Windows can't install extension In-Reply-To: <51C1BB6C.9070206@berlin.de> References: <61e7c3028ece489b84d0c1a91d4d2e15@BLUPR03MB035.namprd03.prod.outlook.com> <51BB3ACE.20702@berlin.de> <51C0DCED.3000100@egenix.com> <51C18D44.3060209@egenix.com> <51C19CC1.50600@python.org> <51C1BB6C.9070206@berlin.de> Message-ID: > > I have no idea why the Windows linker attempts to load a x86 DLL into > > a x64 process. The manifest included with OpenVPN clearly shows that > > the DLL is for a different platform, even though it uses the same > > version as the one listed in the python.exe manifest. > > [snip] > > When Dependency Walker showed a reference from mxDateTime.pyd to > the > 32-bit version of MSVCR90.DLL, it always also showed a reference from > PYTHON27.DLL to the 64-bit version of MSVCR90.DLL. My suspicion is that this is due to OpenVPN adding itself to the PATH variable. The default Python install will put PYTHON27.DLL into System32 (for a 64-bit install), and so it will try and resolve dependencies in the same directory first. This is probably why it finds the correct MSVCR90.DLL (it may also be that the manifest is correct). Since mxDateTime.pyd is not in System32, it will go through the usual resolution process. System32 is supposed to be checked first, so I'm not sure why the correct version is being bypassed, but that may be because of the lack of manifest. I'd guess that it works for most people because by the time Windows reaches the end of its search, the correct DLL is the only/best/most recent option found. That OpenVPN version provides an alternative, and Windows is choosing it and then failing to load because of the platform mismatch. There's not much I can offer by way of solution. Adding a manifest to the pyd 'should' be the 'correct' solution, but I can't guarantee that it won't cause conflicts elsewhere. Adding a copy of MSVCR90.DLL to the pyd directory would also work, but this is probably riskier when it comes to version mismatches. As I understand, OpenVPN has corrected their side of things already, so this may just be an issue to be aware of for when people hit it in the future. Cheers, Steve From jaraco at jaraco.com Wed Jun 19 22:50:03 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Wed, 19 Jun 2013 20:50:03 +0000 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: <3F569970-3D95-4111-92DC-2042DFBC111B@stufft.io> References: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> <82C062DF-6751-41FA-A528-87685558B6D5@jaraco.com> <3F569970-3D95-4111-92DC-2042DFBC111B@stufft.io> Message-ID: <01daddda345a429c9f8171e5ef8f40b3@BLUPR06MB003.namprd06.prod.outlook.com> Donald, What's the next step to support this bootstrap script? Do we need a ticket for PyPI to support this mechanism? How will PyPI resolve the 'latest' virtual link when there are multiple non-hidden versions (as with setuptools 0.7.4 and 0.6c11), or does the uploader make that determination? From: Donald Stufft [mailto:donald at stufft.io] Sent: Monday, 10 June, 2013 14:15 To: Jason R. Coombs Cc: Tres Seaver; Distutils-Sig at Python.Org Subject: Re: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI On Jun 10, 2013, at 2:55 PM, "Jason R. Coombs" > wrote: There seems to be a dominant opinion that the file should be on PyPI, and I don't disagree. One issue is that the file must be mutable. It necessarily contains a reference to the preferred version to be downloaded. That version changes over time. It's typically incremented, but sometimes decremented. It's possible the script could be updated to otherwise discover the most appropriate version. As it's currently implemented however, the file must be expected to change over time. Because the file must change during each release, the system needs to be able to accept an updated version. This is why the Bitbucket Link was referenced, because it can include that version information. I'm open to suggestions on how we can create a perma link to a file that changes over time on PyPI. Put a version in the filename/path, and have a "latest" tag that just 301 redirects to whatever is latest. /bootstrap/setuptools/latest/ez_setup.py -> 301 Redirect to /bootstrap/setuptools/0.7.2/ez_setup.py For now, the most authoritative place I could think of for that file was the bit bucket downloads. Let's identify a mechanism to do the same on PyPI and get that into set of tools for the next release. Sent from my comm ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6572 bytes Desc: not available URL: From ralf at systemexit.de Thu Jun 20 00:58:51 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Thu, 20 Jun 2013 00:58:51 +0200 Subject: [Distutils] Buildout 2.2.0b1 is available fort testing In-Reply-To: (Jim Fulton's message of "Mon, 17 Jun 2013 08:44:30 -0400") References: Message-ID: <87wqpp4sic.fsf@myhost.lan> Jim Fulton writes: > > We'd like to release 2.2 final soon, so your testing is really > helpful. I tested this version and didn't run into any problems, though I didn't test the new stuff. -- Cheers Ralf From pje at telecommunity.com Thu Jun 20 02:48:55 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 19 Jun 2013 20:48:55 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: <01daddda345a429c9f8171e5ef8f40b3@BLUPR06MB003.namprd06.prod.outlook.com> References: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> <82C062DF-6751-41FA-A528-87685558B6D5@jaraco.com> <3F569970-3D95-4111-92DC-2042DFBC111B@stufft.io> <01daddda345a429c9f8171e5ef8f40b3@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: On Wed, Jun 19, 2013 at 4:50 PM, Jason R. Coombs wrote: > > What?s the next step to support this bootstrap script? Can you just upload it with the docs? From donald at stufft.io Thu Jun 20 05:23:53 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 19 Jun 2013 23:23:53 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: <01daddda345a429c9f8171e5ef8f40b3@BLUPR06MB003.namprd06.prod.outlook.com> References: <3AC0DD05-1688-4158-9AEB-00191DC9E26E@stufft.io> <82C062DF-6751-41FA-A528-87685558B6D5@jaraco.com> <3F569970-3D95-4111-92DC-2042DFBC111B@stufft.io> <01daddda345a429c9f8171e5ef8f40b3@BLUPR06MB003.namprd06.prod.outlook.com> Message-ID: <548F7796-9E2C-4C47-A063-3C402A8ED03E@stufft.io> On Jun 19, 2013, at 4:50 PM, "Jason R. Coombs" wrote: > Donald, > What?s the next step to support this bootstrap script? Do we need a ticket for PyPI to support this mechanism? How will PyPI resolve the ?latest? virtual link when there are multiple non-hidden versions (as with setuptools 0.7.4 and 0.6c11), or does the uploader make that determination? > > From: Donald Stufft [mailto:donald at stufft.io] > Sent: Monday, 10 June, 2013 14:15 > To: Jason R. Coombs > Cc: Tres Seaver; Distutils-Sig at Python.Org > Subject: Re: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI > > > On Jun 10, 2013, at 2:55 PM, "Jason R. Coombs" wrote: > > > There seems to be a dominant opinion that the file should be on PyPI, and I don't disagree. > > One issue is that the file must be mutable. It necessarily contains a reference to the preferred version to be downloaded. That version changes over time. It's typically incremented, but sometimes decremented. > > It's possible the script could be updated to otherwise discover the most appropriate version. As it's currently implemented however, the file must be expected to change over time. > > Because the file must change during each release, the system needs to be able to accept an updated version. This is why the Bitbucket Link was referenced, because it can include that version information. > > I'm open to suggestions on how we can create a perma link to a file that changes over time on PyPI. > > Put a version in the filename/path, and have a "latest" tag that just 301 redirects to whatever is latest. > > /bootstrap/setuptools/latest/ez_setup.py -> 301 Redirect to > /bootstrap/setuptools/0.7.2/ez_setup.py > > > > For now, the most authoritative place I could think of for that file was the bit bucket downloads. Let's identify a mechanism to do the same on PyPI and get that into set of tools for the next release. > > Sent from my comm > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA I think we're going to make a small one off web app for uploading it. I'll see if I can whip something up. If it supports: * Uploading files * Versioned Urls * "Latest" url which redirects to the latest version Will that be enough for your uses? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brian at vanguardistas.net Thu Jun 20 09:55:18 2013 From: brian at vanguardistas.net (Brian Sutherland) Date: Thu, 20 Jun 2013 09:55:18 +0200 Subject: [Distutils] Buildout 2.2.0b1 is available fort testing In-Reply-To: References: Message-ID: <20130620075518.GA97000@Brians-MacBook-Air.local> I've been using this the last few days under python 2.7 and virtualenv 1.9.1. No problems whatsoever, apart from having to upgrade setuptools before running the buildout's bootstrap. On Mon, Jun 17, 2013 at 08:44:30AM -0400, Jim Fulton wrote: > Changes: > > - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. > > - Handle both addition and subtraction of elements (+= and -=) on the same key > in the same section. Forward-ported from buildout 1.6. > > - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` > error message being emitted by distribute / setuptools. > > - Extend distutils script generation to support module docstrings and > __future__ imports. > > - Refactored picked versions logic to make it easier to use for plugins. > > - Use ``get_win_launcher`` API to find Windows launcher (falling back to > ``resource_string`` for ``cli.exe``). > > - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` > in current directory during installation (merged from 1.x branch). > > To try it out, copy the new bootstrap file from: > > http://downloads.buildout.org/2.2.0b1/bootstrap.py > > Then run the bootstrap, making sure to include the -t option: > > python bootstrap.py -t > > The -t option is necessary to use the non-final buildout release. > > We'd like to release 2.2 final soon, so your testing is really > helpful. > > Special thanks to Tres Seaver for updating buildout to work with > setuptools 0.7 and for reviewing and merging a number of outstanding > pull requests. Of course, thanks to the folks to contributed the > many other changes in this release. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Brian Sutherland From ncoghlan at gmail.com Thu Jun 20 15:07:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Jun 2013 23:07:57 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) Message-ID: New versions of PEP 426 and PEP 440 are up on python.org: PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ (as before, not including them inline due to sheer length) The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo) There have been several significant changes to the main metadata spec in PEP 426: * Added a basic "jsonschema" based description of the format * The "abbreviated metadata" notion is gone, replaced by "included documents". The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name "Semantic dependencies" (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying "I need this" with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The "*" notation is added to extras to make it easier to say "install all dependencies", along with the "-extra_name" notation to exclude the dependencies for specific extras * The "-" notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * "Install hooks" are now a distinct concept from the still-hypothetical "metabuild" system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed) The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them Other less significant changes to PEP 426 include: * Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the "Release" tweak from PEP 440, and switching "source archive" to refer to the original raw source, while using "sdist" for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these "new" rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the "Rejected Features" appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments) The other PEP 440 changes are also relatively minor: - what were previously called releases are now "final releases", freeing up "release" as a general term for any kind of release (developmental release, pre-release, final release, post release). - "source references" are now "direct references" and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are allowed in a source label The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel, Donald and I missed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Thu Jun 20 15:36:47 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 20 Jun 2013 09:36:47 -0400 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: On Thu, Jun 20, 2013 at 9:07 AM, Nick Coghlan wrote: > New versions of PEP 426 and PEP 440 are up on python.org: > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ > > (as before, not including them inline due to sheer length) > > The bulk of the changes are in this commit: > http://hg.python.org/peps/rev/de305af601fa > (there are a few minor tweaks in subsequent commits to the PEPs repo) > > There have been several significant changes to the main metadata spec > in PEP 426: > > * Added a basic "jsonschema" based description of the format > * The "abbreviated metadata" notion is gone, replaced by "included > documents". The metadata can now list the names of additional files > included in the distinfo directory for the long description, the > license and the change history. The contents are no longer embedded > directly in the metadata (but will be extracted and made available by > PyPI, with the markup format being determined from the file extension, > just as it is by sites like GitHub) > * The dependency system has been redesigned under the name "Semantic > dependencies" (as they now allow a distribution to better say *why* a > dependency is needed, rather than just saying "I need this" with > almost no capacity to say why). There are now five kinds of > dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're > integrated into the extras system so you can easily say you want to > install some, none or all of them. > * The "*" notation is added to extras to make it easier to say > "install all dependencies", along with the "-extra_name" notation to > exclude the dependencies for specific extras > * The "-" notation is added to extras to make it easier to install > *just* a distribution's dependencies, without installing the > distribution itself. > * "Install hooks" are now a distinct concept from the > still-hypothetical "metabuild" system, and place more constraints on > their expected handling (installation tools are also explicitly > permitted to refuse to run them, but doing so is required to fail > noisily rather than silently appearing to succeed) > > The most significant change to PEP 440 is to the handling of > pre-releases: whether or not pre-releases should be accepted by > default is now determined solely by whether or not there is a stable > release that *also* satisfies the version specifier. Reference a > pre-release (or not) now has no effect on whether pre-releases are > considered viable candidates. Pre-releases are now accepted if: > * they're already installed > * there's no other available option > * the installation tool is told specifically to consider them > > Other less significant changes to PEP 426 include: > > * Longer introduction giving more context for the nature and purpose > of the metadata > * Separated various other things out into appendices > * Various tweaks to definitions (including the "Release" tweak from > PEP 440, and switching "source archive" to refer to the original raw > source, while using "sdist" for the Python specific format with the > extra metadata) > * Blanket permission for distribution related online services to > impose additional restrictions to protect the integrity of the service > (such as additional length limits beyond those stated in the PEP). > * Explicitly require UTF-8 encoded JSON for serialisation > * build_label renamed to source_label > * version_url renamed to source_url > * tightened up the validation rules for various fields (including RFC > references where appropriate). Many of these "new" rules are things > projects already comply with (because not complying doesn't work > properly). Including them in the spec is about giving PyPI clear > guidance to enforce them at point of upload, rather than leaving it to > installation tools to try to sort out later. > * a few more additions to the "Rejected Features" appendix (notably, > my rationale for *not* requiring the install hooks to accept arbitrary > keyword arguments) > > The other PEP 440 changes are also relatively minor: > > - what were previously called releases are now "final releases", > freeing up "release" as a general term for any kind of release > (developmental release, pre-release, final release, post release). > - "source references" are now "direct references" and can also refer > to prebuilt wheel files > - automated tools (especially index servers) are given a lot of > freedom to be opinionated about the appropriate uses for direct > references > - a few tweaks to the security guidelines for direct references > - pytz/Olson database version translation is explicitly discussed (add > a leading 0., convert the trailing letter to an incrementing number) > - tighter rules for what characters are allowed in a source label > > The only major remaining open item is the addition of guidelines in > Appendix A for converting legacy metadata to metadata 2.0. I consider > the rest of the spec stable at this point, unless anyone picks up on a > glaring hole in the latest draft that Daniel, Donald and I missed. > > Cheers, > Nick. FYI the tip revision of wheel (bdist_wheel, https://bitbucket.org/dholth/wheel/) produces pymeta.json that is almost up-to-date with this version of the spec, except that it still inlines the description rather than saving it to a separate file. bdist_wheel has always worked by converting setuptools metadata to PEP metadata. From ncoghlan at gmail.com Thu Jun 20 15:42:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Jun 2013 23:42:33 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: On 20 June 2013 23:36, Daniel Holth wrote: > On Thu, Jun 20, 2013 at 9:07 AM, Nick Coghlan wrote: >> The only major remaining open item is the addition of guidelines in >> Appendix A for converting legacy metadata to metadata 2.0. I consider >> the rest of the spec stable at this point, unless anyone picks up on a >> glaring hole in the latest draft that Daniel, Donald and I missed. >> >> Cheers, >> Nick. > > FYI the tip revision of wheel (bdist_wheel, > https://bitbucket.org/dholth/wheel/) produces pymeta.json that is > almost up-to-date with this version of the spec, except that it still > inlines the description rather than saving it to a separate file. > bdist_wheel has always worked by converting setuptools metadata to PEP > metadata. Yeah, pointing at the relevant code in the wheel implementation as one approach that is known to work reasonably well may be the most expedient way of handling that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Thu Jun 20 16:36:22 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 20 Jun 2013 14:36:22 +0000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: <20130620143622.GA5471@merlinux.eu> I still think the testing part of "the interchange format between software publication and integration tools" is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something? best and thanks for your good work, holger On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote: > New versions of PEP 426 and PEP 440 are up on python.org: > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ > > (as before, not including them inline due to sheer length) > > The bulk of the changes are in this commit: > http://hg.python.org/peps/rev/de305af601fa > (there are a few minor tweaks in subsequent commits to the PEPs repo) > > There have been several significant changes to the main metadata spec > in PEP 426: > > * Added a basic "jsonschema" based description of the format > * The "abbreviated metadata" notion is gone, replaced by "included > documents". The metadata can now list the names of additional files > included in the distinfo directory for the long description, the > license and the change history. The contents are no longer embedded > directly in the metadata (but will be extracted and made available by > PyPI, with the markup format being determined from the file extension, > just as it is by sites like GitHub) > * The dependency system has been redesigned under the name "Semantic > dependencies" (as they now allow a distribution to better say *why* a > dependency is needed, rather than just saying "I need this" with > almost no capacity to say why). There are now five kinds of > dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're > integrated into the extras system so you can easily say you want to > install some, none or all of them. > * The "*" notation is added to extras to make it easier to say > "install all dependencies", along with the "-extra_name" notation to > exclude the dependencies for specific extras > * The "-" notation is added to extras to make it easier to install > *just* a distribution's dependencies, without installing the > distribution itself. > * "Install hooks" are now a distinct concept from the > still-hypothetical "metabuild" system, and place more constraints on > their expected handling (installation tools are also explicitly > permitted to refuse to run them, but doing so is required to fail > noisily rather than silently appearing to succeed) > > The most significant change to PEP 440 is to the handling of > pre-releases: whether or not pre-releases should be accepted by > default is now determined solely by whether or not there is a stable > release that *also* satisfies the version specifier. Reference a > pre-release (or not) now has no effect on whether pre-releases are > considered viable candidates. Pre-releases are now accepted if: > * they're already installed > * there's no other available option > * the installation tool is told specifically to consider them > > Other less significant changes to PEP 426 include: > > * Longer introduction giving more context for the nature and purpose > of the metadata > * Separated various other things out into appendices > * Various tweaks to definitions (including the "Release" tweak from > PEP 440, and switching "source archive" to refer to the original raw > source, while using "sdist" for the Python specific format with the > extra metadata) > * Blanket permission for distribution related online services to > impose additional restrictions to protect the integrity of the service > (such as additional length limits beyond those stated in the PEP). > * Explicitly require UTF-8 encoded JSON for serialisation > * build_label renamed to source_label > * version_url renamed to source_url > * tightened up the validation rules for various fields (including RFC > references where appropriate). Many of these "new" rules are things > projects already comply with (because not complying doesn't work > properly). Including them in the spec is about giving PyPI clear > guidance to enforce them at point of upload, rather than leaving it to > installation tools to try to sort out later. > * a few more additions to the "Rejected Features" appendix (notably, > my rationale for *not* requiring the install hooks to accept arbitrary > keyword arguments) > > The other PEP 440 changes are also relatively minor: > > - what were previously called releases are now "final releases", > freeing up "release" as a general term for any kind of release > (developmental release, pre-release, final release, post release). > - "source references" are now "direct references" and can also refer > to prebuilt wheel files > - automated tools (especially index servers) are given a lot of > freedom to be opinionated about the appropriate uses for direct > references > - a few tweaks to the security guidelines for direct references > - pytz/Olson database version translation is explicitly discussed (add > a leading 0., convert the trailing letter to an incrementing number) > - tighter rules for what characters are allowed in a source label > > The only major remaining open item is the addition of guidelines in > Appendix A for converting legacy metadata to metadata 2.0. I consider > the rest of the spec stable at this point, unless anyone picks up on a > glaring hole in the latest draft that Daniel, Donald and I missed. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From jim at zope.com Thu Jun 20 16:37:39 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 20 Jun 2013 10:37:39 -0400 Subject: [Distutils] Heads up: I plan to release buildout 2.2.0 this weekend. Message-ID: Testing has gone well, but I still anticipate a little excitement. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From ncoghlan at gmail.com Thu Jun 20 17:05:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jun 2013 01:05:09 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: <20130620143622.GA5471@merlinux.eu> References: <20130620143622.GA5471@merlinux.eu> Message-ID: On 21 Jun 2013 00:36, "holger krekel" wrote: > > > I still think the testing part of "the interchange format > between software publication and integration tools" is underspecified. > The dependencies alone will not be sufficient to allow the running > of tests in many cases. Or am i missing something? If "setup.py test" works for a distribution today, it will still work under PEP 426. Anything more is deliberately deferred (as noted in the PEP). Cheers, Nick. > > best and thanks for your good work, > > holger > > On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote: > > New versions of PEP 426 and PEP 440 are up on python.org: > > > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ > > > > (as before, not including them inline due to sheer length) > > > > The bulk of the changes are in this commit: > > http://hg.python.org/peps/rev/de305af601fa > > (there are a few minor tweaks in subsequent commits to the PEPs repo) > > > > There have been several significant changes to the main metadata spec > > in PEP 426: > > > > * Added a basic "jsonschema" based description of the format > > * The "abbreviated metadata" notion is gone, replaced by "included > > documents". The metadata can now list the names of additional files > > included in the distinfo directory for the long description, the > > license and the change history. The contents are no longer embedded > > directly in the metadata (but will be extracted and made available by > > PyPI, with the markup format being determined from the file extension, > > just as it is by sites like GitHub) > > * The dependency system has been redesigned under the name "Semantic > > dependencies" (as they now allow a distribution to better say *why* a > > dependency is needed, rather than just saying "I need this" with > > almost no capacity to say why). There are now five kinds of > > dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're > > integrated into the extras system so you can easily say you want to > > install some, none or all of them. > > * The "*" notation is added to extras to make it easier to say > > "install all dependencies", along with the "-extra_name" notation to > > exclude the dependencies for specific extras > > * The "-" notation is added to extras to make it easier to install > > *just* a distribution's dependencies, without installing the > > distribution itself. > > * "Install hooks" are now a distinct concept from the > > still-hypothetical "metabuild" system, and place more constraints on > > their expected handling (installation tools are also explicitly > > permitted to refuse to run them, but doing so is required to fail > > noisily rather than silently appearing to succeed) > > > > The most significant change to PEP 440 is to the handling of > > pre-releases: whether or not pre-releases should be accepted by > > default is now determined solely by whether or not there is a stable > > release that *also* satisfies the version specifier. Reference a > > pre-release (or not) now has no effect on whether pre-releases are > > considered viable candidates. Pre-releases are now accepted if: > > * they're already installed > > * there's no other available option > > * the installation tool is told specifically to consider them > > > > Other less significant changes to PEP 426 include: > > > > * Longer introduction giving more context for the nature and purpose > > of the metadata > > * Separated various other things out into appendices > > * Various tweaks to definitions (including the "Release" tweak from > > PEP 440, and switching "source archive" to refer to the original raw > > source, while using "sdist" for the Python specific format with the > > extra metadata) > > * Blanket permission for distribution related online services to > > impose additional restrictions to protect the integrity of the service > > (such as additional length limits beyond those stated in the PEP). > > * Explicitly require UTF-8 encoded JSON for serialisation > > * build_label renamed to source_label > > * version_url renamed to source_url > > * tightened up the validation rules for various fields (including RFC > > references where appropriate). Many of these "new" rules are things > > projects already comply with (because not complying doesn't work > > properly). Including them in the spec is about giving PyPI clear > > guidance to enforce them at point of upload, rather than leaving it to > > installation tools to try to sort out later. > > * a few more additions to the "Rejected Features" appendix (notably, > > my rationale for *not* requiring the install hooks to accept arbitrary > > keyword arguments) > > > > The other PEP 440 changes are also relatively minor: > > > > - what were previously called releases are now "final releases", > > freeing up "release" as a general term for any kind of release > > (developmental release, pre-release, final release, post release). > > - "source references" are now "direct references" and can also refer > > to prebuilt wheel files > > - automated tools (especially index servers) are given a lot of > > freedom to be opinionated about the appropriate uses for direct > > references > > - a few tweaks to the security guidelines for direct references > > - pytz/Olson database version translation is explicitly discussed (add > > a leading 0., convert the trailing letter to an incrementing number) > > - tighter rules for what characters are allowed in a source label > > > > The only major remaining open item is the addition of guidelines in > > Appendix A for converting legacy metadata to metadata 2.0. I consider > > the rest of the spec stable at this point, unless anyone picks up on a > > glaring hole in the latest draft that Daniel, Donald and I missed. > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdouche at gmail.com Thu Jun 20 17:06:55 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Thu, 20 Jun 2013 17:06:55 +0200 Subject: [Distutils] Heads up: I plan to release buildout 2.2.0 this weekend. In-Reply-To: References: Message-ID: On Thu, Jun 20, 2013 at 4:37 PM, Jim Fulton wrote: > Testing has gone well, but I still anticipate a little excitement. Yeah. Maybe a good idea to wait Setuptools 0.8? -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From jim at zope.com Thu Jun 20 17:08:44 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 20 Jun 2013 11:08:44 -0400 Subject: [Distutils] Heads up: I plan to release buildout 2.2.0 this weekend. In-Reply-To: References: Message-ID: On Thu, Jun 20, 2013 at 11:06 AM, Sebastien Douche wrote: > On Thu, Jun 20, 2013 at 4:37 PM, Jim Fulton wrote: >> Testing has gone well, but I still anticipate a little excitement. > > Yeah. Maybe a good idea to wait Setuptools 0.8? Why? It will pick up setuptools.8 when it becomes available. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From holger at merlinux.eu Thu Jun 20 22:19:17 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 20 Jun 2013 20:19:17 +0000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> Message-ID: <20130620201917.GB5471@merlinux.eu> On Fri, Jun 21, 2013 at 01:05 +1000, Nick Coghlan wrote: > On 21 Jun 2013 00:36, "holger krekel" wrote: > > > > > > I still think the testing part of "the interchange format > > between software publication and integration tools" is underspecified. > > The dependencies alone will not be sufficient to allow the running > > of tests in many cases. Or am i missing something? > > If "setup.py test" works for a distribution today, it will still work under > PEP 426. Anything more is deliberately deferred (as noted in the PEP). I am a little confused. Early in the PEP you say: The published metadata for distributions SHOULD allow integrators, with the aid of build and integration tools, to: ... * if supported by the distribution, run the distribution's automatic test suite on an installed instance of the distribution but a few sentences later: The current iteration of the metadata relies on the distutils commands system to support other necessary integration and deployment activities: ... python setup.py test: run the distribution's test suite on a built (but not yet installed) distribution And i am not sure what exactly PEP426 defines wrt to testing now. I saw the "test_requires" and "test_may_requires" fields and thought they were part of an idea already how to configure running of tests. But is there anything more in the current PEP426 that i missed? To help with my confusion, could you maybe describe how "python setup.py test" is to interoperate (if at all) with the new metadata format as is? thanks, holger > Cheers, > Nick. > > > > > best and thanks for your good work, > > > > holger > > > > On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote: > > > New versions of PEP 426 and PEP 440 are up on python.org: > > > > > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > > PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ > > > > > > (as before, not including them inline due to sheer length) > > > > > > The bulk of the changes are in this commit: > > > http://hg.python.org/peps/rev/de305af601fa > > > (there are a few minor tweaks in subsequent commits to the PEPs repo) > > > > > > There have been several significant changes to the main metadata spec > > > in PEP 426: > > > > > > * Added a basic "jsonschema" based description of the format > > > * The "abbreviated metadata" notion is gone, replaced by "included > > > documents". The metadata can now list the names of additional files > > > included in the distinfo directory for the long description, the > > > license and the change history. The contents are no longer embedded > > > directly in the metadata (but will be extracted and made available by > > > PyPI, with the markup format being determined from the file extension, > > > just as it is by sites like GitHub) > > > * The dependency system has been redesigned under the name "Semantic > > > dependencies" (as they now allow a distribution to better say *why* a > > > dependency is needed, rather than just saying "I need this" with > > > almost no capacity to say why). There are now five kinds of > > > dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're > > > integrated into the extras system so you can easily say you want to > > > install some, none or all of them. > > > * The "*" notation is added to extras to make it easier to say > > > "install all dependencies", along with the "-extra_name" notation to > > > exclude the dependencies for specific extras > > > * The "-" notation is added to extras to make it easier to install > > > *just* a distribution's dependencies, without installing the > > > distribution itself. > > > * "Install hooks" are now a distinct concept from the > > > still-hypothetical "metabuild" system, and place more constraints on > > > their expected handling (installation tools are also explicitly > > > permitted to refuse to run them, but doing so is required to fail > > > noisily rather than silently appearing to succeed) > > > > > > The most significant change to PEP 440 is to the handling of > > > pre-releases: whether or not pre-releases should be accepted by > > > default is now determined solely by whether or not there is a stable > > > release that *also* satisfies the version specifier. Reference a > > > pre-release (or not) now has no effect on whether pre-releases are > > > considered viable candidates. Pre-releases are now accepted if: > > > * they're already installed > > > * there's no other available option > > > * the installation tool is told specifically to consider them > > > > > > Other less significant changes to PEP 426 include: > > > > > > * Longer introduction giving more context for the nature and purpose > > > of the metadata > > > * Separated various other things out into appendices > > > * Various tweaks to definitions (including the "Release" tweak from > > > PEP 440, and switching "source archive" to refer to the original raw > > > source, while using "sdist" for the Python specific format with the > > > extra metadata) > > > * Blanket permission for distribution related online services to > > > impose additional restrictions to protect the integrity of the service > > > (such as additional length limits beyond those stated in the PEP). > > > * Explicitly require UTF-8 encoded JSON for serialisation > > > * build_label renamed to source_label > > > * version_url renamed to source_url > > > * tightened up the validation rules for various fields (including RFC > > > references where appropriate). Many of these "new" rules are things > > > projects already comply with (because not complying doesn't work > > > properly). Including them in the spec is about giving PyPI clear > > > guidance to enforce them at point of upload, rather than leaving it to > > > installation tools to try to sort out later. > > > * a few more additions to the "Rejected Features" appendix (notably, > > > my rationale for *not* requiring the install hooks to accept arbitrary > > > keyword arguments) > > > > > > The other PEP 440 changes are also relatively minor: > > > > > > - what were previously called releases are now "final releases", > > > freeing up "release" as a general term for any kind of release > > > (developmental release, pre-release, final release, post release). > > > - "source references" are now "direct references" and can also refer > > > to prebuilt wheel files > > > - automated tools (especially index servers) are given a lot of > > > freedom to be opinionated about the appropriate uses for direct > > > references > > > - a few tweaks to the security guidelines for direct references > > > - pytz/Olson database version translation is explicitly discussed (add > > > a leading 0., convert the trailing letter to an incrementing number) > > > - tighter rules for what characters are allowed in a source label > > > > > > The only major remaining open item is the addition of guidelines in > > > Appendix A for converting legacy metadata to metadata 2.0. I consider > > > the rest of the spec stable at this point, unless anyone picks up on a > > > glaring hole in the latest draft that Daniel, Donald and I missed. > > > > > > Cheers, > > > Nick. > > > > > > -- > > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > _______________________________________________ > > > Distutils-SIG maillist - Distutils-SIG at python.org > > > http://mail.python.org/mailman/listinfo/distutils-sig > > > From dholth at gmail.com Thu Jun 20 22:54:52 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 20 Jun 2013 16:54:52 -0400 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: <20130620201917.GB5471@merlinux.eu> References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On Thu, Jun 20, 2013 at 4:19 PM, holger krekel wrote: > On Fri, Jun 21, 2013 at 01:05 +1000, Nick Coghlan wrote: >> On 21 Jun 2013 00:36, "holger krekel" wrote: >> > >> > >> > I still think the testing part of "the interchange format >> > between software publication and integration tools" is underspecified. >> > The dependencies alone will not be sufficient to allow the running >> > of tests in many cases. Or am i missing something? >> >> If "setup.py test" works for a distribution today, it will still work under >> PEP 426. Anything more is deliberately deferred (as noted in the PEP). > > I am a little confused. Early in the PEP you say: > > The published metadata for distributions SHOULD allow integrators, > with the aid of build and integration tools, to: > > ... > * if supported by the distribution, run the distribution's automatic > test suite on an installed instance of the distribution > > but a few sentences later: > > The current iteration of the metadata relies on the distutils > commands system to support other necessary integration and > deployment activities: > > ... > python setup.py test: run the distribution's test suite on a built > (but not yet installed) distribution > > And i am not sure what exactly PEP426 defines wrt to testing now. > I saw the "test_requires" and "test_may_requires" fields and thought > they were part of an idea already how to configure running of tests. > But is there anything more in the current PEP426 that i missed? > > To help with my confusion, could you maybe describe how "python setup.py > test" is to interoperate (if at all) with the new metadata format as is? > > thanks, > holger "python setup.py test" does not change at all as a result of this work. The only interoperability that will happen is that an external test runner will better be able to ask an installer to install those dependencies before attempting to run the test, since installs-as-side-effects-of-setup.py are not so great. Later setup.py will be replaceable, and /that/ specification will define a standard entry point for testing. These metadata PEPs are mostly about the installer, database of installed files, and package indexes. From p.f.moore at gmail.com Thu Jun 20 23:40:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 20 Jun 2013 22:40:15 +0100 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On 20 June 2013 21:54, Daniel Holth wrote: > These metadata PEPs are > mostly about the installer, database of installed files, and package > indexes. > In essence (and I freely admit I don't fully understand the details yet myself) I think the idea is that PEPs 426 and 440 define a set of data that can be published by packages. The *how* of publishing that data is still to be defined - the idea is that there will be the appropriate metadata JSON files available, but no tools currently produce that format. Equally, installers like pip will be able to read the metadata and ensure that dependencies are installed appropriately (for example) but again, that will be a future change to pip. For now, the PEPs define a standard for tools to *use*, but leave the creation of such tools (or modification of existing tools) as a later step. I don't think the PEPs say anything about a plan for implementing the spec (although I haven't read the PEPs in detail yet, so I may have missed something) but my assumption is that it'll go something like this: 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. 2. Pypi will be changed to publish metadata from pymeta.json, when it is present, which tools like pip can consume (see later). 3. Wheel will put pymeta.json into wheels, from the pymeta.json created by (1) and likely by converting legacy egg-info metadata as a fallback. Wheel will also write pymeta.json on install. 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install, and use installed pymeta.json (again probably falling back to legacy formats where pymeta.json isn't present). 5. Distlib will follow suit. The big gap here is older versions of Python. Maybe a setuptools plugin to generate the new metadata in sdists would be viable. Maybe package writers could provide pymeta.json by hand, although I seriously doubt many will bother. How close am I to reality here? Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jun 21 00:29:54 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 20 Jun 2013 18:29:54 -0400 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On Jun 20, 2013, at 5:40 PM, Paul Moore wrote: > On 20 June 2013 21:54, Daniel Holth wrote: > These metadata PEPs are > mostly about the installer, database of installed files, and package > indexes. > > In essence (and I freely admit I don't fully understand the details yet myself) I think the idea is that PEPs 426 and 440 define a set of data that can be published by packages. The *how* of publishing that data is still to be defined - the idea is that there will be the appropriate metadata JSON files available, but no tools currently produce that format. Equally, installers like pip will be able to read the metadata and ensure that dependencies are installed appropriately (for example) but again, that will be a future change to pip. > > For now, the PEPs define a standard for tools to *use*, but leave the creation of such tools (or modification of existing tools) as a later step. > > I don't think the PEPs say anything about a plan for implementing the spec (although I haven't read the PEPs in detail yet, so I may have missed something) but my assumption is that it'll go something like this: > > 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. sdist 2.0 is not part of the PEP afaik. There will be a separate PEP for that. Wheel's will be the only thing generating these files to begin with. > 2. Pypi will be changed to publish metadata from pymeta.json, when it is present, which tools like pip can consume (see later). > 3. Wheel will put pymeta.json into wheels, from the pymeta.json created by (1) and likely by converting legacy egg-info metadata as a fallback. Wheel will also write pymeta.json on install. > 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install, and use installed pymeta.json (again probably falling back to legacy formats where pymeta.json isn't present). > 5. Distlib will follow suit. > > The big gap here is older versions of Python. Maybe a setuptools plugin to generate the new metadata in sdists would be viable. Maybe package writers could provide pymeta.json by hand, although I seriously doubt many will bother. > > How close am I to reality here? > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Basically the plan with PEP426 is to come up with the "installer" side of things. There's places that things like building (going from sdist to wheel) or development (going from checkout to sdist) can be incorporated. And some thoughts have been made for it but afaik the primary focus has been on making this possible with wheels while leaving us a path to doing the same with sdists. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri Jun 21 00:51:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jun 2013 08:51:50 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: <20130620201917.GB5471@merlinux.eu> References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On 21 Jun 2013 06:19, "holger krekel" wrote: > > On Fri, Jun 21, 2013 at 01:05 +1000, Nick Coghlan wrote: > > On 21 Jun 2013 00:36, "holger krekel" wrote: > > > > > > > > > I still think the testing part of "the interchange format > > > between software publication and integration tools" is underspecified. > > > The dependencies alone will not be sufficient to allow the running > > > of tests in many cases. Or am i missing something? > > > > If "setup.py test" works for a distribution today, it will still work under > > PEP 426. Anything more is deliberately deferred (as noted in the PEP). > > I am a little confused. Early in the PEP you say: > > The published metadata for distributions SHOULD allow integrators, > with the aid of build and integration tools, to: > > ... > * if supported by the distribution, run the distribution's automatic > test suite on an installed instance of the distribution It's just an editing error - this was explicitly rejected (and is listed as a rejected feature), I just missed one of the references that was in the previous draft. Cheers, Nick. > > but a few sentences later: > > The current iteration of the metadata relies on the distutils > commands system to support other necessary integration and > deployment activities: > > ... > python setup.py test: run the distribution's test suite on a built > (but not yet installed) distribution > > And i am not sure what exactly PEP426 defines wrt to testing now. > I saw the "test_requires" and "test_may_requires" fields and thought > they were part of an idea already how to configure running of tests. > But is there anything more in the current PEP426 that i missed? > > To help with my confusion, could you maybe describe how "python setup.py > test" is to interoperate (if at all) with the new metadata format as is? > > thanks, > holger > > > > Cheers, > > Nick. > > > > > > > > best and thanks for your good work, > > > > > > holger > > > > > > On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote: > > > > New versions of PEP 426 and PEP 440 are up on python.org: > > > > > > > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > > > > PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ > > > > > > > > (as before, not including them inline due to sheer length) > > > > > > > > The bulk of the changes are in this commit: > > > > http://hg.python.org/peps/rev/de305af601fa > > > > (there are a few minor tweaks in subsequent commits to the PEPs repo) > > > > > > > > There have been several significant changes to the main metadata spec > > > > in PEP 426: > > > > > > > > * Added a basic "jsonschema" based description of the format > > > > * The "abbreviated metadata" notion is gone, replaced by "included > > > > documents". The metadata can now list the names of additional files > > > > included in the distinfo directory for the long description, the > > > > license and the change history. The contents are no longer embedded > > > > directly in the metadata (but will be extracted and made available by > > > > PyPI, with the markup format being determined from the file extension, > > > > just as it is by sites like GitHub) > > > > * The dependency system has been redesigned under the name "Semantic > > > > dependencies" (as they now allow a distribution to better say *why* a > > > > dependency is needed, rather than just saying "I need this" with > > > > almost no capacity to say why). There are now five kinds of > > > > dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're > > > > integrated into the extras system so you can easily say you want to > > > > install some, none or all of them. > > > > * The "*" notation is added to extras to make it easier to say > > > > "install all dependencies", along with the "-extra_name" notation to > > > > exclude the dependencies for specific extras > > > > * The "-" notation is added to extras to make it easier to install > > > > *just* a distribution's dependencies, without installing the > > > > distribution itself. > > > > * "Install hooks" are now a distinct concept from the > > > > still-hypothetical "metabuild" system, and place more constraints on > > > > their expected handling (installation tools are also explicitly > > > > permitted to refuse to run them, but doing so is required to fail > > > > noisily rather than silently appearing to succeed) > > > > > > > > The most significant change to PEP 440 is to the handling of > > > > pre-releases: whether or not pre-releases should be accepted by > > > > default is now determined solely by whether or not there is a stable > > > > release that *also* satisfies the version specifier. Reference a > > > > pre-release (or not) now has no effect on whether pre-releases are > > > > considered viable candidates. Pre-releases are now accepted if: > > > > * they're already installed > > > > * there's no other available option > > > > * the installation tool is told specifically to consider them > > > > > > > > Other less significant changes to PEP 426 include: > > > > > > > > * Longer introduction giving more context for the nature and purpose > > > > of the metadata > > > > * Separated various other things out into appendices > > > > * Various tweaks to definitions (including the "Release" tweak from > > > > PEP 440, and switching "source archive" to refer to the original raw > > > > source, while using "sdist" for the Python specific format with the > > > > extra metadata) > > > > * Blanket permission for distribution related online services to > > > > impose additional restrictions to protect the integrity of the service > > > > (such as additional length limits beyond those stated in the PEP). > > > > * Explicitly require UTF-8 encoded JSON for serialisation > > > > * build_label renamed to source_label > > > > * version_url renamed to source_url > > > > * tightened up the validation rules for various fields (including RFC > > > > references where appropriate). Many of these "new" rules are things > > > > projects already comply with (because not complying doesn't work > > > > properly). Including them in the spec is about giving PyPI clear > > > > guidance to enforce them at point of upload, rather than leaving it to > > > > installation tools to try to sort out later. > > > > * a few more additions to the "Rejected Features" appendix (notably, > > > > my rationale for *not* requiring the install hooks to accept arbitrary > > > > keyword arguments) > > > > > > > > The other PEP 440 changes are also relatively minor: > > > > > > > > - what were previously called releases are now "final releases", > > > > freeing up "release" as a general term for any kind of release > > > > (developmental release, pre-release, final release, post release). > > > > - "source references" are now "direct references" and can also refer > > > > to prebuilt wheel files > > > > - automated tools (especially index servers) are given a lot of > > > > freedom to be opinionated about the appropriate uses for direct > > > > references > > > > - a few tweaks to the security guidelines for direct references > > > > - pytz/Olson database version translation is explicitly discussed (add > > > > a leading 0., convert the trailing letter to an incrementing number) > > > > - tighter rules for what characters are allowed in a source label > > > > > > > > The only major remaining open item is the addition of guidelines in > > > > Appendix A for converting legacy metadata to metadata 2.0. I consider > > > > the rest of the spec stable at this point, unless anyone picks up on a > > > > glaring hole in the latest draft that Daniel, Donald and I missed. > > > > > > > > Cheers, > > > > Nick. > > > > > > > > -- > > > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > > _______________________________________________ > > > > Distutils-SIG maillist - Distutils-SIG at python.org > > > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 21 01:24:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jun 2013 09:24:24 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On 21 Jun 2013 07:40, "Paul Moore" wrote: > > On 20 June 2013 21:54, Daniel Holth wrote: >> >> These metadata PEPs are >> mostly about the installer, database of installed files, and package >> indexes. > > > In essence (and I freely admit I don't fully understand the details yet myself) I think the idea is that PEPs 426 and 440 define a set of data that can be published by packages. The *how* of publishing that data is still to be defined - the idea is that there will be the appropriate metadata JSON files available, but no tools currently produce that format. Equally, installers like pip will be able to read the metadata and ensure that dependencies are installed appropriately (for example) but again, that will be a future change to pip. > > For now, the PEPs define a standard for tools to *use*, but leave the creation of such tools (or modification of existing tools) as a later step. > > I don't think the PEPs say anything about a plan for implementing the spec (although I haven't read the PEPs in detail yet, so I may have missed something) Right, the general implementation plan isn't part of the PEP, aside from the big note at the beginning about all the *other* PEPs that will be needed to finish bootstrapping this thing. > but my assumption is that it'll go something like this: > > 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. The initial vehicles for migration will be Daniel's bdist_wheel command and upgrades to setuptools. Standard library support may not land for 3.4, but we will at least have the pip bootstrapping mechanism in place (which in turn will allow retrieval of setuptools). Until the updated packaging ecosystem stabilises, adopting the standard library's update rate would be a bad idea. Besides, a solution that doesn't work back as far Python 2.6 faces significant barriers to community adoption. > 2. Pypi will be changed to publish metadata from pymeta.json, when it is present, which tools like pip can consume (see later). /me lets cat out of bag Donald's actually working on a second generation PyPI design that uses PEP 426 as its underlying data model for distributions. It will probably include the capability to derive 2.0 metadata from legacy metadata, so installers can benefit from the new mechanisms at install time, even for existing distributions. Many of the security constraints and backwards compatibility tweaks in the PEP actually come from that design work. With the spec now stabilised, we'll likely flip the public bit on that repo soon, and start working on some concrete plans to spin up a test instance alongside the existing PyPI service :) Along with Daniel ensuring that bdist_wheel can emit the format, Donald's work on the server side has been a key part of keeping the metadata spec grounded in reality. Once we flip the public bit on the repo, the next generation PyPI will serve alongside Daniel's existing wheel project as the reference implementations for the spec. > 3. Wheel will put pymeta.json into wheels, from the pymeta.json created by (1) and likely by converting legacy egg-info metadata as a fallback. Wheel will also write pymeta.json on install. On install, it should be covered by the existing "copy the dist-info contents from the wheel" behaviour, but that's the general idea. sdist 2.0 will just deprecate PKG-INFO and replace it with a self identifying wheel-style dist-info subdirectory. > 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install, and use installed pymeta.json (again probably falling back to legacy formats where pymeta.json isn't present). > 5. Distlib will follow suit. Yup (although *definitely* falling back to legacy formats). > > The big gap here is older versions of Python. Maybe a setuptools plugin to generate the new metadata in sdists would be viable. Maybe package writers could provide pymeta.json by hand, although I seriously doubt many will bother. > > How close am I to reality here? Close, except that this is explicitly an "ecosystem first" plan, with the standard library playing catch up later. That's the big reason I sought (and received) agreement at this year's language summit for us to discuss and approve packaging related PEPs right here on distutils-sig rather than having to repost them to python-dev for ratification. The only "must have" on the list for 3.4 is the pip bootstrapping, and Richard and I hope to get that squared away at the PyCon AU sprints next month. Cheers, Nick. > > Paul > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Jun 21 11:52:29 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Jun 2013 10:52:29 +0100 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On 21 June 2013 00:24, Nick Coghlan wrote: > > How close am I to reality here? > > Close, except that this is explicitly an "ecosystem first" plan, with the > standard library playing catch up later. That's the big reason I sought > (and received) agreement at this year's language summit for us to discuss > and approve packaging related PEPs right here on distutils-sig rather than > having to repost them to python-dev for ratification. > > The only "must have" on the list for 3.4 is the pip bootstrapping, and > Richard and I hope to get that squared away at the PyCon AU sprints next > month. > Thanks. My remaining concern is the "chicken and egg" question of how this much richer metadata will get populated - things like dependency metadata is only as useful as the accuracy of the information. Some of it can be translated from existing legacy formats (notably setuptools) and hopefully once Appendix A is completed, we'll have a better understanding of how that will work, and how complete it will be. As regards the actual PEP contents, my thoughts are basically: - I'm +1 on the PEP as a whole. Most of it seems to be to be entirely reasonable and an important step forward - I don't have much personal use for the more complex items like semantic dependencies, extras, hooks, etc, but I see no particular problems with them - Thinking about what I *do* use, the most glaring omission is a specific "supports Python 3" indicator. I know it's available in the classifiers, and it may have already been discussed, but is there realistically any benefit in formalising this? The environment markers section covers this somewhat, but is it within the remit of the PEP to recommend how (or even when) packages should declare what major versions of Python they support? (And should installers be warning about attempts to install packages that don't support the relevant Python major version?) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 21 13:35:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jun 2013 21:35:41 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On 21 Jun 2013 19:52, "Paul Moore" wrote: > > On 21 June 2013 00:24, Nick Coghlan wrote: >> >> > How close am I to reality here? >> >> Close, except that this is explicitly an "ecosystem first" plan, with the standard library playing catch up later. That's the big reason I sought (and received) agreement at this year's language summit for us to discuss and approve packaging related PEPs right here on distutils-sig rather than having to repost them to python-dev for ratification. >> >> The only "must have" on the list for 3.4 is the pip bootstrapping, and Richard and I hope to get that squared away at the PyCon AU sprints next month. > > Thanks. My remaining concern is the "chicken and egg" question of how this much richer metadata will get populated - things like dependency metadata is only as useful as the accuracy of the information. Some of it can be translated from existing legacy formats (notably setuptools) and hopefully once Appendix A is completed, we'll have a better understanding of how that will work, and how complete it will be. The initial metadata will come from existing setuptools based requires and install_requires definitions (thanks to easy_install and pip, there's a lot of good dependency metadata available already). Having PyPI extract and publish that info (rather than having to download each distribution to read it) will already be a substantial step up from the status quo. Over the next few years, I expect professional developers and Linux distribution packagers to help improve the metadata of their existing dependencies. At the moment, we *can't* easily propose upstream patches for dependency issues, but one of my goals with the PEP is to make fully automated generation of distro policy compliant distro specific packages feasible (at least for packages which don't have any special requirements). At that point, the distro's preferred "upstream first" approach becomes feasible for metadata fixes as well, rather than being limited to source code changes. > As regards the actual PEP contents, my thoughts are basically: > > - I'm +1 on the PEP as a whole. Most of it seems to be to be entirely reasonable and an important step forward > > - I don't have much personal use for the more complex items like semantic dependencies, extras, hooks, etc, but I see no particular problems with them Yeah, they're definitely power tools. I see several of them as the software distribution equivalent of metaclasses: if you're wondering whether or not you need them, you don't need them. On the other hand, when you *do* need them, you'd be completely stuck if they didn't exist. I'm hopeful that install hooks, metadata extensions and direct references will provide sufficient "escape valves" to handle some seriously esoteric use cases without any further changes to the specification. > - Thinking about what I *do* use, the most glaring omission is a specific "supports Python 3" indicator. I know it's available in the classifiers, and it may have already been discussed, but is there realistically any benefit in formalising this? The environment markers section covers this somewhat, but is it within the remit of the PEP to recommend how (or even when) packages should declare what major versions of Python they support? (And should installers be warning about attempts to install packages that don't support the relevant Python major version?) The quick compatibility check is actually part of the wheel file naming specification (it's covered by the compatibility tags defined in PEP 425). For PEP 426, I originally retained the separate "Requires-Python" field, but eventually realised it was redundant given the "supports_environments" field (http://www.python.org/dev/peps/pep-0426/#supports-environments) and the "python_version" and "python_full_version" variables in environment markers. It gets a little clumsy if you want to express a Python version constraint *and* a platform constraint (due to the implicit or), but the current approach does have the virtue of handling cases where version requirements differ between platforms. For example, if you need atomic renaming capabilities, then they've been available on POSIX systems since pretty much forever, but on Windows you need Python 3.3 so you can have access to os.replace. In the current PEP, that might look like: "supports_environments": ["sys_platform != 'win32' and python_version >= 2.6", "sys_platform == 'win32' and python_version >= 3.3"] If your package was source compatible between 2.6, 2.7 and 3.3+ you might say: "supports_environments": ["python_version == 2.6", "python_version == 2.7", "python_version >= 3.3"] However, looking at the PEP, I realise I didn't actually update the remaining text to reflect that realisation - I just deleted the old text that was no longer applicable. I'll add an example that shows using "supports_environments" to specify a minimum Python version in a way installation tools are expected to check. I'll also make it clearer that "supports_environments" describes what the *sdist* supports. Generating a wheel file may limit it further (as indicated by the compatibility tags). Cheers, Nick. From p.f.moore at gmail.com Fri Jun 21 14:21:59 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Jun 2013 13:21:59 +0100 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: On 21 June 2013 12:35, Nick Coghlan wrote: > The quick compatibility check is actually part of the wheel file > naming specification (it's covered by the compatibility tags defined > in PEP 425). > Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec? > For PEP 426, I originally retained the separate > "Requires-Python" field, but eventually realised it was redundant > given the "supports_environments" field > (http://www.python.org/dev/peps/pep-0426/#supports-environments) and > the "python_version" and "python_full_version" variables in > environment markers. It gets a little clumsy if you want to express a > Python version constraint *and* a platform constraint (due to the > implicit or), but the current approach does have the virtue of > handling cases where version requirements differ between platforms. > For example, if you need atomic renaming capabilities, then they've > been available on POSIX systems since pretty much forever, but on > Windows you need Python 3.3 so you can have access to os.replace. In > the current PEP, that might look like: > > "supports_environments": ["sys_platform != 'win32' and > python_version >= 2.6", "sys_platform == 'win32' and python_version >= > 3.3"] > Yes, but that's very much the "power user" end of the spectrum, as you mentioned before. > If your package was source compatible between 2.6, 2.7 and 3.3+ you might > say: > > "supports_environments": ["python_version == 2.6", "python_version > == 2.7", "python_version >= 3.3"] > More often than not, my thinking is more at the level of "did I take the time to make the code dual-version compatible" - things like putting brackets round print arguments. Or simply "will I accept bug reports for both Python 2 and Python 3, even though I only really test on Python3". The supports_environments tag seems too precise for that sort of very broad brush approach. However, looking at the PEP, I realise I didn't actually update the > remaining text to reflect that realisation - I just deleted the old > text that was no longer applicable. I'll add an example that shows > using "supports_environments" to specify a minimum Python version in a > way installation tools are expected to check. > > I'll also make it clearer that "supports_environments" describes what > the *sdist* supports. Generating a wheel file may limit it further (as > indicated by the compatibility tags). > That sounds good. I'd suggest examples like twisted (doesn't support Python 3 yet), virtualenv (supports Python 2.6 or greater, including 3.x) and maybe something that is purely for Python 3. I'd also like to see a paragraph explaining that installers SHOULD at least respect python version specs (even if they don't handle the full environment marker spec). I'm not sure how practical it would be to require that sort of partial support, though, which is my main reservation about using the environment spec for this. (I think respecting language version specs is important, but the full environment spec is no more than a nice to have). Also, it seems to me that wheel could use the Python version spec to generate appropriate compatibility tags. Again, that's not possible for tags in their full generality, but is possible for "Python version only" tags. And again, I'd like to see something saying that wheel generators should do this (otherwise, there's a duplication somewhere as this information would need to be specified elsewhere in a generator-specific manner, like the setup.cfg extension I mentioned above). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Fri Jun 21 15:30:36 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 21 Jun 2013 15:30:36 +0200 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: <2248FD5D-396E-46F8-A2A5-DBBB34520BCE@mac.com> On 21 Jun, 2013, at 14:21, Paul Moore wrote: > On 21 June 2013 12:35, Nick Coghlan wrote: > The quick compatibility check is actually part of the wheel file > naming specification (it's covered by the compatibility tags defined > in PEP 425). > > Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec? No, but it should be in the spec/documentation for the tool that builds sdists or wheels. That is, when metadata 2.0 support is added to distutils/setuptools/... it should document how the metadata is configured. Ronald From ronaldoussoren at mac.com Fri Jun 21 15:42:20 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 21 Jun 2013 15:42:20 +0200 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> Message-ID: <1696B28F-3D12-4429-A57B-DDF398A3C7A5@mac.com> On 21 Jun, 2013, at 1:24, Nick Coghlan wrote: > > > > 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. > > The initial vehicles for migration will be Daniel's bdist_wheel command and upgrades to setuptools. > > Standard library support may not land for 3.4, but we will at least have the pip bootstrapping mechanism in place (which in turn will allow retrieval of setuptools). What would be needed to land support for 3.4? Is it "just" a matter of pushing distlib to the stdlib and updating distutils (add a way to configure 2.0 metadata, add bdist_wheel and update sdist), or are there other other issues? Well, other than finalizing the PEP and its implementation in distlib? Ronald From holger at merlinux.eu Fri Jun 21 15:49:41 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 21 Jun 2013 13:49:41 +0000 Subject: [Distutils] devpi-0.9.2: uploading existing archives files Message-ID: <20130621134941.GI5471@merlinux.eu> devpi, the caching pypi server and its optional upload/test/install helper tool, just got a devpi-0.9.2 release. See the full updated docs here: http://doc.devpi.net Apart from some streamlining, there is a new upload option:: devpi upload --from-dir path/to/dir [--only-latest] which uploads existing release files to a devpi index, optionally only the latest versions. This is achieved with the help of ``pkginfo`` extracting PKGINFO files and, for the few where it's missing, using the filename to determine name/version data. Thanks to Anthon van der Neut for his related PRs. See below for a more detailed changelog. best, holger 0.9.2 (compared to 0.9.1) ---------------------------- server: - fix /USER/INDEXNAME root views to contain only latest in-stage packages - make +api calls return bases so that "devpi use" can show them client: - introduce "devpi upload --fromdir" for uploading archives in bulk mode, thanks to Anthon van der Neut for helping with this! (resolved issue5) - improve automatic server handling via "devpi use" - for "devpi server" you now need to specify "--log" to see log lines - make "devpi use" also show base indexes by default - fix issue4: auto-server data stored in non-dot dir From p.f.moore at gmail.com Fri Jun 21 16:10:38 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Jun 2013 15:10:38 +0100 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: <2248FD5D-396E-46F8-A2A5-DBBB34520BCE@mac.com> References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> <2248FD5D-396E-46F8-A2A5-DBBB34520BCE@mac.com> Message-ID: On 21 June 2013 14:30, Ronald Oussoren wrote: > > Yes, I was thinking more of the sdist side - and also of how a pure > Python package can specify whether separate wheels are needed for Python 2 > and Python 3, for example. There's a wheel-specific setup.cfg extension for > this, but should it be in the metadata spec? > > No, but it should be in the spec/documentation for the tool that builds > sdists or wheels. That is, when metadata 2.0 support is added to > distutils/setuptools/... it should document how the metadata is configured. So a package author creates a package that only works with Python 3, builds a sdist using a tool-specific option that states this, and that data isn't recorded in the sdist so that whoever uses the sdist to build a wheel has to find that out elsewhere and re-specify it to build a wheel? That sounds wrong, unless I'm misunderstanding you... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 21 16:52:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Jun 2013 00:52:37 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> <2248FD5D-396E-46F8-A2A5-DBBB34520BCE@mac.com> Message-ID: On 22 June 2013 00:10, Paul Moore wrote: > On 21 June 2013 14:30, Ronald Oussoren wrote: >> >> > Yes, I was thinking more of the sdist side - and also of how a pure >> > Python package can specify whether separate wheels are needed for Python 2 >> > and Python 3, for example. There's a wheel-specific setup.cfg extension for >> > this, but should it be in the metadata spec? >> >> No, but it should be in the spec/documentation for the tool that builds >> sdists or wheels. That is, when metadata 2.0 support is added to >> distutils/setuptools/... it should document how the metadata is configured. > > > So a package author creates a package that only works with Python 3, builds > a sdist using a tool-specific option that states this, and that data isn't > recorded in the sdist so that whoever uses the sdist to build a wheel has to > find that out elsewhere and re-specify it to build a wheel? That sounds > wrong, unless I'm misunderstanding you... This is a build problem. The metadata 2.0 spec *deliberately* avoids trying to solve build problems, it merely leaves the door open to attempting to solve them in the future. It *may* make sense for a build tool to check the "supports_environment" flags for implied options when building a wheel, but that's up to the build tool developer. Designing a metabuild system that is actually better than the setup.py based command line API is an incredibly hard problem, and I'm not yet convinced it's a problem we really need to solve. Killing "setup.py install"? Sure, that definitely needs to happen and decoupling the intergrator's choice of install tool from the developer's choice of build tool is the main aim of metadata 2.0. Killing "setup.py dist_info/sdist/bdist_wheel/test", though? I'm *not* convinced that needs to happen, since there's no way to decouple of project's build system from itself and projects like d2to1 show that creating a setup.py based shim for an alternate build system doesn't need to be *that* painful, particularly once there are well documented standard archive formats to target. Even if a setup.py replacement on the build side does happen eventually, it's a long way down the priority list. In the meantime, once sdist 2.0 is officially defined in a PEP, build tools will be able to define their own metadata extensions to record whatever they want in the sdist files. They'll also be free to add arbitrary files to the dist-info directory and have them propagate through the wheel files and all the way to the installation database. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Jun 21 16:57:53 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Jun 2013 15:57:53 +0100 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> <2248FD5D-396E-46F8-A2A5-DBBB34520BCE@mac.com> Message-ID: On 21 June 2013 15:52, Nick Coghlan wrote: > In the meantime, once sdist 2.0 is officially defined in a PEP, build > tools will be able to define their own metadata extensions to record > whatever they want in the sdist files. They'll also be free to add > arbitrary files to the dist-info directory and have them propagate > through the wheel files and all the way to the installation database. > Ah, OK. I see how that would work now. Thanks for the clarification. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Sat Jun 22 15:13:49 2013 From: holger at merlinux.eu (holger krekel) Date: Sat, 22 Jun 2013 13:13:49 +0000 Subject: [Distutils] tox-1.5.0: uses setuptools, whitelisting commands, MIT license Message-ID: <20130622131349.GM5471@merlinux.eu> tox-1.5.0 is a new release of the virtualenv-managing generic Python test runner. It comes with a few fixes and improvements (see below), and is now released under the MIT license -- prior versions used the GPL2. The main change is that tox now by default creates virtualenv's with setuptools, after the merge of distribute (thanks Jason R. Coombs). See docs at: http://tox.testrun.org and install with: pip install -U tox Thanks to Lukasz Balcerzak, Krisztian Fekete for PRs, Armin Ronacher and others for issue reporting, holger 1.5.0 ----------------- - fix issue104: use setuptools by default, instead of distribute, now that setuptools has distribute merged. - make sure test commands are searched first in the virtualenv - re-fix issue2 - add whitelist_externals to be used in ``[testenv*]`` sections, allowing to avoid warnings for commands such as ``make``, used from the commands value. - fix issue97 - allow substitutions to reference from other sections (thanks Krisztian Fekete) - fix issue92 - fix {envsitepackagesdir} to actually work again - show (test) command that is being executed, thanks Lukasz Balcerzak - re-license tox to MIT license - depend on virtualenv-1.9.1 - rename README.txt to README.rst to make bitbucket happier From anthon.van.der.neut at googlemail.com Sat Jun 22 07:56:42 2013 From: anthon.van.der.neut at googlemail.com (Anthon van der Neut) Date: Fri, 21 Jun 2013 22:56:42 -0700 (PDT) Subject: [Distutils] devpi-0.9.2: uploading existing archives files In-Reply-To: <20130621134941.GI5471@merlinux.eu> References: <20130621134941.GI5471@merlinux.eu> Message-ID: <165292fb-3695-4f04-8b8e-66466dc66d00@googlegroups.com> Holger, you should probably update the CHANGELOG ( 0.9.2 client section) to read --from-dir instead of the old format --fromdir Anthon On Friday, 21 June 2013 15:49:41 UTC+2, holger krekel wrote: > > > devpi, the caching pypi server and its optional upload/test/install helper > tool, just got a devpi-0.9.2 release. See the full updated docs here: > > http://doc.devpi.net > > Apart from some streamlining, there is a new upload option:: > > devpi upload --from-dir path/to/dir [--only-latest] > > which uploads existing release files to a devpi index, optionally only > the latest versions. This is achieved with the help of ``pkginfo`` > extracting PKGINFO files and, for the few where it's missing, using the > filename to determine name/version data. Thanks to Anthon van der Neut > for his related PRs. > > See below for a more detailed changelog. > > best, > holger > > 0.9.2 (compared to 0.9.1) > ---------------------------- > > server: > > - fix /USER/INDEXNAME root views to contain only latest in-stage packages > > - make +api calls return bases so that "devpi use" can show them > > client: > > - introduce "devpi upload --fromdir" for uploading archives in bulk > mode, thanks to Anthon van der Neut for helping with this! > (resolved issue5) > > - improve automatic server handling via "devpi use" > > - for "devpi server" you now need to specify "--log" to see log lines > > - make "devpi use" also show base indexes by default > > - fix issue4: auto-server data stored in non-dot dir > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf at systemexit.de Sat Jun 22 23:08:52 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Sat, 22 Jun 2013 23:08:52 +0200 Subject: [Distutils] [ANN] pypiserver 1.1.2 - minimal private pypi server Message-ID: <871u7tx38b.fsf@myhost.lan> Hi, I've just uploaded pypiserver 1.1.2 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. https://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver Changes in this version ----------------------- - fix "pypi-server -U" stable/unstable detection, i.e. do not accidentally update to unstable packages -- Cheers Ralf From dholth at gmail.com Sun Jun 23 03:23:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Sat, 22 Jun 2013 21:23:24 -0400 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: <20130620143622.GA5471@merlinux.eu> <20130620201917.GB5471@merlinux.eu> <2248FD5D-396E-46F8-A2A5-DBBB34520BCE@mac.com> Message-ID: Tip revision of bdist_wheel ( https://bitbucket.org/dholth/wheel/ ) supports this revision of the spec, with the description in a separate file. It also has better support for representing conditional dependencies in the metadata. On Fri, Jun 21, 2013 at 10:57 AM, Paul Moore wrote: > On 21 June 2013 15:52, Nick Coghlan wrote: >> >> In the meantime, once sdist 2.0 is officially defined in a PEP, build >> tools will be able to define their own metadata extensions to record >> whatever they want in the sdist files. They'll also be free to add >> arbitrary files to the dist-info directory and have them propagate >> through the wheel files and all the way to the installation database. > > > Ah, OK. I see how that would work now. Thanks for the clarification. > Paul > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From techtonik at gmail.com Sun Jun 23 07:33:36 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 23 Jun 2013 08:33:36 +0300 Subject: [Distutils] Fixing PyPI download stats with real-time log analysis (Was: PyPI Download Counts) In-Reply-To: References: Message-ID: On Fri, Jun 14, 2013 at 5:05 PM, anatoly techtonik wrote: > Could you, please, share the log format+example, so that we can experiment > with it? > ping -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Sun Jun 23 08:56:57 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sat, 22 Jun 2013 23:56:57 -0700 Subject: [Distutils] Fixing PyPI download stats with real-time log analysis (Was: PyPI Download Counts) In-Reply-To: References: Message-ID: <10391719-FE9A-4FD9-B8EB-C6FE9AFA62D7@coderanger.net> On Jun 22, 2013, at 10:33 PM, anatoly techtonik wrote: > On Fri, Jun 14, 2013 at 5:05 PM, anatoly techtonik wrote: > Could you, please, share the log format+example, so that we can experiment with it? > > ping Additional assistance is not required for this project. Thank you for your interest, and stay tuned for future updates. --Noah From ncoghlan at gmail.com Sun Jun 23 09:05:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Jun 2013 17:05:09 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: On 20 June 2013 23:07, Nick Coghlan wrote: > New versions of PEP 426 and PEP 440 are up on python.org: > > PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ > PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ > > (as before, not including them inline due to sheer length) > > The bulk of the changes are in this commit: > http://hg.python.org/peps/rev/de305af601fa > (there are a few minor tweaks in subsequent commits to the PEPs repo) PEP 426 has now been updated based on the feedback on this thread (and to handle some "todo" items that were pending anyway): http://hg.python.org/peps/rev/3f733fe7c06c * Adds implementation_name and implementation_version marker variables * Expands on the expected use cases for supports_environments * References wheel and warehouse from legacy metadata appendix * Drops most of the Sphinx notes (moving some to Rejected Features) I forgot to mention it in the commit message, but it also expands on some of the possible use cases for metadata extensions, such as a build tool storing the default build options for wheel creation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Jun 23 13:09:30 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 23 Jun 2013 12:09:30 +0100 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: On 23 June 2013 08:05, Nick Coghlan wrote: > PEP 426 has now been updated based on the feedback on this thread (and > to handle some "todo" items that were pending anyway): > http://hg.python.org/peps/rev/3f733fe7c06c > Sorry I didn't think of this sooner, but I don't see in the PEP a brief summary of precisely what metadata is required. From the "Core Metadata" section I infer that a project with *only* Name and Version (and Metadata-Version, which will be supplied by the build tool) is valid. That's fine, but notably more permissive than earlier versions of the spec. One thing I see is "Index servers MAY require that this field be provided" for some metadata (e.g., Summary") which is fine, I guess, but implies that somewhere, the rules for PyPI should be documented. Also, is it true that build tools MAY require certain fields to be provided? I'm pretty sure setup.py sdist does at present. My logic here is that previously I would go to the metadata PEP to check precisely what fields I need when building my project. Now, I'm not sure where I'd go. (Note that most of my projects are internal only, so it's actually the extra distutils/setuptools requirements that interest me more than those of PyPI...) I can imagine the answer would be "the documentation of the individual tool" - but in reality said documentation often isn't present (possibly because earlier metadata PEPs documented this information centrally) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jun 23 14:53:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Jun 2013 22:53:26 +1000 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: On 23 Jun 2013 21:09, "Paul Moore" wrote: > > On 23 June 2013 08:05, Nick Coghlan wrote: >> >> PEP 426 has now been updated based on the feedback on this thread (and >> to handle some "todo" items that were pending anyway): >> http://hg.python.org/peps/rev/3f733fe7c06c > > > Sorry I didn't think of this sooner, but I don't see in the PEP a brief summary of precisely what metadata is required. From the "Core Metadata" section I infer that a project with *only* Name and Version (and Metadata-Version, which will be supplied by the build tool) is valid. That's fine, but notably more permissive than earlier versions of the spec. That's deliberate - the mandatory fields are just those where dependency analysis can't work if they're missing. > One thing I see is "Index servers MAY require that this field be provided" for some metadata (e.g., Summary") which is fine, I guess, but implies that somewhere, the rules for PyPI should be documented. I think I'll just move the summary back up into the core metadata. You can't make a sane index server without it, and it used to be mandatory, so "maybe mandatory, maybe not" just makes things more complicated than they need to be rather than simplifying anything. > Also, is it true that build tools MAY require certain fields to be provided? I'm pretty sure setup.py sdist does at present. > > My logic here is that previously I would go to the metadata PEP to check precisely what fields I need when building my project. Now, I'm not sure where I'd go. (Note that most of my projects are internal only, so it's actually the extra distutils/setuptools requirements that interest me more than those of PyPI...) I can imagine the answer would be "the documentation of the individual tool" - but in reality said documentation often isn't present (possibly because earlier metadata PEPs documented this information centrally) I expect tools to come into line with PEP 426 on that front over time. Those that still generate legacy metadata will likely follow the legacy requirements, and for tools with stronger opinions about what should be provided, it will be up to the authors to document that properly. Contributions to the Python packaging user guide (a shared introduction to setuptools and pip) are also more than welcome (see http://python-packaging-user-guide.readthedocs.org). Marcus has made a good start based on the distribute guide (with less opinionated hyperbole), and I expect it to settle down to something quite usable once pip 1.4 is released and it can simply direct people towards setuptools 0.7+ as the default build tool (plus "wheel" for bdist_wheel support) and pip 1.4+ as the default installer (including support for installation from wheels, and automatic use of cached local builds for installation from source). We really don't want end users needing to read PEPs just to get started. Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Sun Jun 23 15:18:56 2013 From: aclark at aclark.net (Alex Clark) Date: Sun, 23 Jun 2013 13:18:56 +0000 (UTC) Subject: [Distutils] Fixing PyPI download stats with real-time log analysis (Was: PyPI Download Counts) References: Message-ID: anatoly techtonik gmail.com> writes: > > On Fri, Jun 14, 2013 at 5:05 PM, anatoly techtonik gmail.com> wrote:Could you, please, share the log format+example, so that we can experiment with it? > > > ping +1. I'm wavering between: accepting they are gone forever, and hoping they get fixed ASAP :-). I'm also willing to help so if you need anything please let me know. From p.f.moore at gmail.com Sun Jun 23 15:22:02 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 23 Jun 2013 14:22:02 +0100 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: On 23 June 2013 13:53, Nick Coghlan wrote: > I think I'll just move the summary back up into the core metadata. You > can't make a sane index server without it, and it used to be mandatory, so > "maybe mandatory, maybe not" just makes things more complicated than they > need to be rather than simplifying anything. > Yes, it's the "maybe mandatory, depending on your tools" state that's annoying. If summary was the only one of them, then this change makes sense. Thanks. Older metadata versions made things like Download-URL mandatory, which was a pain for internal projects that didn't actually *have* a download URL, just a file on the software server or something (and no, file URLs don't count :-)) I guess they all become optional under Metadata 2.0 (as per the "Changes from Previous Versions" appendix). > I expect tools to come into line with PEP 426 on that front over time. > Those that still generate legacy metadata will likely follow the legacy > requirements, and for tools with stronger opinions about what should be > provided, it will be up to the authors to document that properly. > OK, that's cool. As tools move towards *validating* to the 2.0 spec, the problem goes away. It's just that there's no way to make older tools validate to the new spec, so there's a transition here which we have to live with (as opposed to *generating* metadata 2.0, which can be handled by converters). > Contributions to the Python packaging user guide (a shared introduction to > setuptools and pip) are also more than welcome (see > http://python-packaging-user-guide.readthedocs.org). Marcus has made a > good start based on the distribute guide (with less opinionated hyperbole), > and I expect it to settle down to something quite usable once pip 1.4 is > released and it can simply direct people towards setuptools 0.7+ as the > default build tool (plus "wheel" for bdist_wheel support) and pip 1.4+ as > the default installer (including support for installation from wheels, and > automatic use of cached local builds for installation from source). > Yes, I should have a look at that. > We really don't want end users needing to read PEPs just to get started. > Agreed, but again there's not much else to work with at the moment :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Sun Jun 23 15:36:28 2013 From: aclark at aclark.net (Alex Clark) Date: Sun, 23 Jun 2013 13:36:28 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Fixing_PyPI_download_stats_with_real-time_l?= =?utf-8?q?og=09analysis_=28Was=3A_PyPI_Download_Counts=29?= References: <10391719-FE9A-4FD9-B8EB-C6FE9AFA62D7@coderanger.net> Message-ID: Hi Noah, Noah Kantrowitz coderanger.net> writes: > > > On Jun 22, 2013, at 10:33 PM, anatoly techtonik wrote: > > > On Fri, Jun 14, 2013 at 5:05 PM, anatoly techtonik gmail.com> wrote: > > Could you, please, share the log format+example, so that we can experiment with it? > > > > ping > > Additional assistance is not required for this project. Thank you for your interest, and stay tuned for > future updates. WAT. Can you explain why "additional assistance is not required for this project"? AFAICT: - Shit is broken (i.e. download counts) - At first, there was no plans to fix it (which I accepted as a cost to enable the CDN) - Now there are plans to fix it, but no additional assistance is needed? If that were the case I'd expect download counts to be fixed. Or at the very least, I'd expect a "no thanks we don't need help, but here is where we are at [ explanation of situation ]". Not "we don't need help, we'll be in touch". I think the latter is a disservice to the community, which (I assume) you especially don't want to be responsible for given the tremendously positive service you've done by enabling the CDN in the first place.) We get it: download counts were not as important as CDN. But they *are* important, so let's keep talking about how to fix them. Thank you, Alex > > --Noah > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From dholth at gmail.com Sun Jun 23 16:06:13 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 23 Jun 2013 10:06:13 -0400 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: The short list of required fields is also based on what is actually in common use. If you compare to 1.1 instead of 1.2 it's a lot closer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jun 23 17:11:56 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Jun 2013 11:11:56 -0400 Subject: [Distutils] Fixing PyPI download stats with real-time log analysis (Was: PyPI Download Counts) In-Reply-To: References: <10391719-FE9A-4FD9-B8EB-C6FE9AFA62D7@coderanger.net> Message-ID: <9D98E581-B557-4478-B0D2-1D93C4060DC3@stufft.io> On Jun 23, 2013, at 9:36 AM, Alex Clark wrote: > Hi Noah, > > > Noah Kantrowitz coderanger.net> writes: > >> >> >> On Jun 22, 2013, at 10:33 PM, anatoly techtonik wrote: >> >>> On Fri, Jun 14, 2013 at 5:05 PM, anatoly techtonik > gmail.com> wrote: >>> Could you, please, share the log format+example, so that we can > experiment with it? >>> >>> ping >> >> Additional assistance is not required for this project. Thank you for your > interest, and stay tuned for >> future updates. > > > WAT. > > > Can you explain why "additional assistance is not required for this > project"? AFAICT: > > - Shit is broken (i.e. download counts) > - At first, there was no plans to fix it (which I accepted as a cost to > enable the CDN) > - Now there are plans to fix it, but no additional assistance is needed? If > that were the case I'd expect download counts to be fixed. Or at the very > least, I'd expect a "no thanks we don't need help, but here is where we are > at [ explanation of situation ]". Not "we don't need help, we'll be in > touch". I think the latter is a disservice to the community, which (I > assume) you especially don't want to be responsible for given the > tremendously positive service you've done by enabling the CDN in the first > place.) > > > We get it: download counts were not as important as CDN. But they *are* > important, so let's keep talking about how to fix them. https://en.wikipedia.org/wiki/Brooks%27_law To further expand. Right now the biggest issue preventing the CDN download counts is a less than ideal configuration on the VM hosting PyPI. I've been working on (and almost have completed) a Chef cookbook that will allow us to easily deploy PyPI to a second VM (and gracefully do the switch over) with minimal downtime. However there is an issue with launching VM's at the moment that OSUOL is looking into which means even with a completed cookbook we couldn't launch a VM at the moment anyways. If we wanted to correct the configuration in place it would require downtime. > > > Thank you, > > > Alex > > > >> >> --Noah >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> >> > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Sun Jun 23 19:29:24 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 23 Jun 2013 17:29:24 +0000 (UTC) Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) References: Message-ID: Nick Coghlan gmail.com> writes: > > New versions of PEP 426 and PEP 440 are up on python.org: > Thanks for the update; I'm in the process of updating distlib to provide support for the updated spec. The tip version of distlib provides good coverage for all except the very latest updates to the spec, and they should be included soon. It's a very minor point, but I noticed that all the keys of the main metadata dictionary and all sub-dictionaries are lower case with underscore, except for the project_urls dictionary which has keys like "Home", "Documentation" etc. Is there any reason for this stylistic departure in just this part of the spec? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sun Jun 23 19:52:13 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 23 Jun 2013 17:52:13 +0000 (UTC) Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) References: Message-ID: Nick Coghlan gmail.com> writes: > * References wheel and warehouse from legacy metadata appendix It may also be worth mentioning that: * distlib supports converting legacy metadata to the new format; it's as simple as: from distlib.metadata import Metadata metadata = Metadata(path='PKG-INFO') metadata.write(path='pymeta.json') * Metadata extracted from setup.py, which includes full dependency information, is available through distlib's locate function. This includes setuptools' newest format for extras, which has support for for environment markers. For example: from distlib.locators import locate dist = locate('setuptools') dist.metadata.write('pymeta.json') Then pymeta.json contains (some fields omitted for brevity, but the dependency stuff is there): { "version": "0.7.4", "name": "setuptools", "metadata_version": "2.0", "run_may_require": [ { "environment": "sys_platform=='win32'", "dependencies": [ "wincertstore (== 0.1)" ], "extra": "ssl" }, { "environment": "sys_platform=='win32' and python_version=='2.4'", "dependencies": [ "ctypes (== 1.0.2)" ], "extra": "ssl" }, { "dependencies": [ "certifi (== 0.0.8)" ], "extra": "certs" }, { "environment": "python_version in '2.4, 2.5'", "dependencies": [ "ssl (== 1.16)" ], "extra": "ssl" } ], "extras": [ "certs", "ssl" ], } I would welcome feedback from anyone who cares to clone/sync the distlib repository. A release of distlib with this functionality will occur once incorporation of the latest spec changes occurs, and I'm comfortable that distlib is reasonably stable following these changes. Additional testing by others can speed up getting to that point :-) Regards, Vinay Sajip From dholth at gmail.com Sun Jun 23 23:04:18 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 23 Jun 2013 17:04:18 -0400 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: On Sun, Jun 23, 2013 at 1:52 PM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> * References wheel and warehouse from legacy metadata appendix > > It may also be worth mentioning that: > > * distlib supports converting legacy metadata to the new format; it's as > simple as: > > from distlib.metadata import Metadata > metadata = Metadata(path='PKG-INFO') > metadata.write(path='pymeta.json') > > * Metadata extracted from setup.py, which includes full dependency > information, is available through distlib's locate function. This > includes setuptools' newest format for extras, which has support for > for environment markers. For example: > > from distlib.locators import locate > dist = locate('setuptools') > dist.metadata.write('pymeta.json') > > Then pymeta.json contains (some fields omitted for brevity, but the > dependency stuff is there): > > { > "version": "0.7.4", > "name": "setuptools", > "metadata_version": "2.0", > "run_may_require": [ > { > "environment": "sys_platform=='win32'", > "dependencies": [ > "wincertstore (== 0.1)" > ], > "extra": "ssl" > }, > { > "environment": "sys_platform=='win32' and python_version=='2.4'", > "dependencies": [ > "ctypes (== 1.0.2)" > ], > "extra": "ssl" > }, > { > "dependencies": [ > "certifi (== 0.0.8)" > ], > "extra": "certs" > }, > { > "environment": "python_version in '2.4, 2.5'", > "dependencies": [ > "ssl (== 1.16)" > ], > "extra": "ssl" > } > ], > "extras": [ > "certs", > "ssl" > ], > } > > I would welcome feedback from anyone who cares to clone/sync the distlib > repository. A release of distlib with this functionality will occur once > incorporation of the latest spec changes occurs, and I'm comfortable that > distlib is reasonably stable following these changes. Additional testing by > others can speed up getting to that point :-) > > Regards, > > Vinay Sajip Here is the setuptools extras+conditionals format. It just separates them with a colon: extras_require = { "ssl:sys_platform=='win32'": "wincertstore==0.1", "ssl:sys_platform=='win32' and python_version=='2.4'": "ctypes==1.0.2", "ssl:python_version in '2.4, 2.5'":"ssl==1.16", "certs": "certifi==0.0.8", }, From vinay_sajip at yahoo.co.uk Mon Jun 24 01:57:59 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 23 Jun 2013 23:57:59 +0000 (UTC) Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) References: Message-ID: Daniel Holth gmail.com> writes: > Here is the setuptools extras+conditionals format. It just separates > them with a colon: Thanks, but I've already seen that. The example I gave showed how that was migrated to the PEP 426 format as pymeta.json :-) Did I miss anything? Regards, Vinay Sajip From dholth at gmail.com Mon Jun 24 02:01:51 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 23 Jun 2013 20:01:51 -0400 Subject: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440) In-Reply-To: References: Message-ID: No I just forgot the syntax so I looked it up. On Jun 23, 2013 7:58 PM, "Vinay Sajip" wrote: > Daniel Holth gmail.com> writes: > > > > Here is the setuptools extras+conditionals format. It just separates > > them with a colon: > > Thanks, but I've already seen that. The example I gave showed how that was > migrated to the PEP 426 format as pymeta.json :-) > > Did I miss anything? > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jun 24 13:56:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Jun 2013 21:56:01 +1000 Subject: [Distutils] Some more updates to PEP 426 Message-ID: Based on the last thread, a few more tweaks to PEP 426: * distlib added as a reference impl in Appendix A * summary field is mandatory again * explicitly advised against using line breaks in the summary and license fields * cleaned up description of the standard setup.py based build system (unified the two separate lists into one, added the missing reference to "build_ext --inplace" that is an implicit precursor to "setup.py test" * explicit advised against invoking "setup.py install" for installation from source (instead recommending building a wheel archive and installing from that - this is what pip will do when caching builds anyway, so we may as well be explicit that if developers want code to run on target systems, they're going to have to adopt metadata 2.0 install hooks) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tseaver at palladion.com Mon Jun 24 19:14:07 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 24 Jun 2013 13:14:07 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: Message-ID: <51C87E5F.6050706@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/2013 12:22 PM, Jason R. Coombs wrote: > Additionally, I released Distribute 0.7 to PyPI. This means that users > who have Distribute 0.6.x can upgrade to setuptools 0.7 by simply > invoking `easy_install -U distribute` (or `easy_install > "distribute>=0.7"`). Windows users should use `easy_install-script -U > distribute` (to avoid "file in use" errors on easy_install.exe). Jason, the cheeseshop doesn't show a distribute 0.7 release. We have folks who can't easily test the new version of zc.buildout because they are stuck on a setuptools-denying version of distribute. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHIfl8ACgkQ+gerLs4ltQ7PaQCgruExzWYv6I9qw6LQSx5RwnzH hPQAoI1Ta2lImcTdgHzq52U3N2eKv5o3 =kyNO -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Jun 24 23:11:16 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 24 Jun 2013 17:11:16 -0400 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: References: <51C87E5F.6050706@palladion.com> Message-ID: <51C8B5F4.6000006@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/24/2013 04:44 PM, Jason R. Coombs wrote: > I had to pull distribute 0.7 because older versions of pip couldn't > handle the upgrade. You mean 'pip install -U distribute' fails with distribute 0.7? This works for me in a brand-new virtualenv w/ distribute: $ bin/pip install \ --find-links=https://bitbucket.org/pypa/setuptools/downloads/ \ -U distribute > I'll be re-adding distribute 0.7 back to PyPI following the new > release of pip. Can you test by pointing to the bitbucket downloads in > the short term? > > Can you tell me about what issues the buildout users are experiencing? > Are they trying to upgrade a buildout from distribute to setuptools? > Or is there an issue with creating new buildouts against the latest > setuptools? No issue with the latest setuptools. The "prevent setuptools installation at all costs" behavior of distribute 0.6.x kills the buildout (or its bootstrap) which tries to install setuptools 0.7.2. If there was a clean way to upgrade from distribute 0.6 before running the bootstrap, that would be enough: but then, that was what distribute 0.7 was supposed to do. :( > I'm hesitant to release Distribute 0.7 because even the latest version > will break the 'easy_install' scripts for users who upgrade via pip. > Actually, on second thought, pip users might consider that a feature. pip users can't spell easy_install, so you should be safe. :) > If there's going to be a re-release of distribute 0.7, it'll need to > be coordinated with the pip and virtualenv releases, which we're > planning for the weekend after next (Jul 6). I know that's a long > time, but I'm hoping there's a suitable workaround for buildout. If > not, then getting an updated distribute (and possibly pip) out sooner > (even without immediate) virtualenv support might be possible. Where are the problems pip users reported with distribute 0.7 archived? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHItfQACgkQ+gerLs4ltQ5PWQCdHenszGzMEbOXgzmHyIKK/EwW X4oAnRE7uAYmBQE97gabYiWFbjRA1Egz =/k4B -----END PGP SIGNATURE----- From qwcode at gmail.com Tue Jun 25 00:08:24 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 24 Jun 2013 15:08:24 -0700 Subject: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI In-Reply-To: <51C8B5F4.6000006@palladion.com> References: <51C87E5F.6050706@palladion.com> <51C8B5F4.6000006@palladion.com> Message-ID: > If there's going to be a re-release of distribute 0.7, it'll need to > > be coordinated with the pip and virtualenv releases, which we're > > planning for the weekend after next (Jul 6). I know that's a long > > time, but I'm hoping there's a suitable workaround for buildout. If > > not, then getting an updated distribute (and possibly pip) out sooner > > (even without immediate) virtualenv support might be possible. > > Where are the problems pip users reported with distribute 0.7 archived? > > issue and discussion: https://github.com/pypa/pip/issues/986 associated pull: https://github.com/pypa/pip/pull/992 Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From yury at shurup.com Mon Jun 24 17:07:49 2013 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 24 Jun 2013 17:07:49 +0200 Subject: [Distutils] Better way of forcing distutils to use platlib location? Message-ID: <1372086469.2844.49.camel@newpride> Hi, I'm working on Python bindings for a C++ software, which are basically a thin wrapper around a (binary) extension. The extension itself is compiled, linked and installed using autotools. The main motivation for doing it this way stems from the fact that previous attempts to hack distutils classes resulted in a monster script bigger than the autotools build system for the C++ package itself. Still, I'd like to keep using distutils to install the pure Python part. Now that I'm not using the Extension class of distutils to build the extension, however, it doesn't know that the Python stuff should go to platlib along with the extension shared object. So my question is, what is the least horrible way to force distutils to install a pure Python package into platlib instead of purelib? I was thinking of monkeypatching get_python_lib(), but there must certainly be a better solution to the problem that I have overlooked! Thanks and please kindly CC me on replies, -- Sincerely yours, Yury V. Zaytsev From dholth at gmail.com Tue Jun 25 14:26:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 25 Jun 2013 08:26:24 -0400 Subject: [Distutils] Better way of forcing distutils to use platlib location? In-Reply-To: <1372086469.2844.49.camel@newpride> References: <1372086469.2844.49.camel@newpride> Message-ID: On Mon, Jun 24, 2013 at 11:07 AM, Yury V. Zaytsev wrote: > Hi, > > I'm working on Python bindings for a C++ software, which are basically a > thin wrapper around a (binary) extension. The extension itself is > compiled, linked and installed using autotools. > > The main motivation for doing it this way stems from the fact that > previous attempts to hack distutils classes resulted in a monster script > bigger than the autotools build system for the C++ package itself. > > Still, I'd like to keep using distutils to install the pure Python part. > Now that I'm not using the Extension class of distutils to build the > extension, however, it doesn't know that the Python stuff should go to > platlib along with the extension shared object. > > So my question is, what is the least horrible way to force distutils to > install a pure Python package into platlib instead of purelib? > > I was thinking of monkeypatching get_python_lib(), but there must > certainly be a better solution to the problem that I have overlooked! > > Thanks and please kindly CC me on replies, > > -- > Sincerely yours, > Yury V. Zaytsev It looks like you may be able to pass distclass=... to setup.py with a distutils.dist.Distribution subclass that overrides is_pure() to return False. From cedric.dsm at gmail.com Tue Jun 25 14:11:31 2013 From: cedric.dsm at gmail.com (=?iso-8859-1?Q?C=E9dric_de_Saint_Martin?=) Date: Tue, 25 Jun 2013 14:11:31 +0200 Subject: [Distutils] (no subject) In-Reply-To: <51C8B903.3000604@palladion.com> References: <07F4617E-7743-478F-8102-8B0ED0A74FD7@gmail.com> <51C8B903.3000604@palladion.com> Message-ID: <32B7F9FB-1B1A-4097-B0FA-A26283A793D6@gmail.com> On 24 juin 2013, at 23:24, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/24/2013 05:06 PM, Reinout van Rees wrote: > >> Uninstalling distribute? Nice for those with the distribute they got >> from their Ubuntu package... > > Those users need to be lobbying their distro to release a > 'python-setuptools' 0.7.x package to get them out of the dead end they > are in. I'm sorry but... Do we really need the distributions to be updated to have "normal users" able to use buildout? I don't want to start a flame war, but the release/merge of setuptools/distribute gave many users bad headaches. The recent activity on setuptools is really great. The merge was the best thing that could have happened. But honestly, our team already having lost several days of work because of the 0.7 release that didn't work (because of distribute or many different things), when you say "please wait 4 months that a new major release of Ubuntu is out (because I don't think they'll switch so easily to new setuptools in a minor release) so that your users can use buildout", i'm getting mad. I haven't said anything until now, because I know it's *hard* to make this work and be released as expected. But please, take into consideration that the vast majority of setuptools/distribute users don't give a shit about this (most of them don't even know what it is), but they just want something that works without having to loose half a day. Ideally, it should be completely transparent for the user. In the worst case, he should have to do something trivial. I still feel that there is no good reason not to release (for real, i.e on the cheeseshop and so on) a dummy distribute 0.7 depending on setuptools 0.7. Maybe I missed a point? Would it break even more things? Cedric > >> Distribute deprecated and must-be-eradicated? Well, the same was said >> of the virtually unmaintained setuptools. Remember the >> use-pip-plus-distribute soviet-style propaganda images? > > I remember the propaganda, but never bought it. People who *do* buy into > propaganda have some reponsibility for the consequences, no? > >> I don't know why distribute users should bear the pain caused by the >> bad original maintainership of setuptools... > > I think you are missing the point: a distribute 0.7 meta-release is > *for* those users, because it allows them to upgrade from distribute > 0.6.x into the brave new post-merge world. Jason says that there are > some reports that pip cannot upgrade to distribute 0.7.x: I cannot > reproduce that myself. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlHIuQMACgkQ+gerLs4ltQ6q7QCffHjoMMTnnM/D9HFRCtydjWX2 > +IsAn26vmw71lGX/Q8HexN+xnGqntSnx > =6gux > -----END PGP SIGNATURE----- > > -- > You received this message because you are subscribed to the Google Groups "Buildout Development" group. > To unsubscribe from this group and stop receiving emails from it, send an email to buildout-development+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > From ncoghlan at gmail.com Tue Jun 25 16:22:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Jun 2013 00:22:46 +1000 Subject: [Distutils] (no subject) In-Reply-To: <32B7F9FB-1B1A-4097-B0FA-A26283A793D6@gmail.com> References: <07F4617E-7743-478F-8102-8B0ED0A74FD7@gmail.com> <51C8B903.3000604@palladion.com> <32B7F9FB-1B1A-4097-B0FA-A26283A793D6@gmail.com> Message-ID: On 25 Jun 2013 22:47, "C?dric de Saint Martin" wrote: > > > On 24 juin 2013, at 23:24, Tres Seaver wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > On 06/24/2013 05:06 PM, Reinout van Rees wrote: > > > >> Uninstalling distribute? Nice for those with the distribute they got > >> from their Ubuntu package... > > > > Those users need to be lobbying their distro to release a > > 'python-setuptools' 0.7.x package to get them out of the dead end they > > are in. > I'm sorry but... Do we really need the distributions to be updated to have "normal users" able to use buildout? > I don't want to start a flame war, but the release/merge of setuptools/distribute gave many users bad headaches. > > The recent activity on setuptools is really great. The merge was the best thing that could have happened. But honestly, our team already having lost several days of work because of the 0.7 release that didn't work (because of distribute or many different things), when you say "please wait 4 months that a new major release of Ubuntu is out (because I don't think they'll switch so easily to new setuptools in a minor release) so that your users can use buildout", i'm getting mad. > > I haven't said anything until now, because I know it's *hard* to make this work and be released as expected. But please, take into consideration that the vast majority of setuptools/distribute users don't give a shit about this (most of them don't even know what it is), but they just want something that works without having to loose half a day. Ideally, it should be completely transparent for the user. In the worst case, he should have to do something trivial. > > I still feel that there is no good reason not to release (for real, i.e on the cheeseshop and so on) a dummy distribute 0.7 depending on setuptools 0.7. Maybe I missed a point? Would it break even more things? That was done originally, but it broke pip badly and hence was withdrawn almost immediately. Once pip has been updated to handle it, it will be restored to PyPI. Cheers, Nick. > > Cedric > > > > >> Distribute deprecated and must-be-eradicated? Well, the same was said > >> of the virtually unmaintained setuptools. Remember the > >> use-pip-plus-distribute soviet-style propaganda images? > > > > I remember the propaganda, but never bought it. People who *do* buy into > > propaganda have some reponsibility for the consequences, no? > > > >> I don't know why distribute users should bear the pain caused by the > >> bad original maintainership of setuptools... > > > > I think you are missing the point: a distribute 0.7 meta-release is > > *for* those users, because it allows them to upgrade from distribute > > 0.6.x into the brave new post-merge world. Jason says that there are > > some reports that pip cannot upgrade to distribute 0.7.x: I cannot > > reproduce that myself. > > > > > > Tres. > > - -- > > =================================================================== > > Tres Seaver +1 540-429-0999 tseaver at palladion.com > > Palladion Software "Excellence by Design" http://palladion.com > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v1.4.11 (GNU/Linux) > > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > > > iEYEARECAAYFAlHIuQMACgkQ+gerLs4ltQ6q7QCffHjoMMTnnM/D9HFRCtydjWX2 > > +IsAn26vmw71lGX/Q8HexN+xnGqntSnx > > =6gux > > -----END PGP SIGNATURE----- > > > > -- > > You received this message because you are subscribed to the Google Groups "Buildout Development" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to buildout-development+unsubscribe at googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Jun 25 18:09:56 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 25 Jun 2013 12:09:56 -0400 Subject: [Distutils] Better way of forcing distutils to use platlib location? In-Reply-To: <1372176115.6605.56.camel@newpride> References: <1372086469.2844.49.camel@newpride> <1372176115.6605.56.camel@newpride> Message-ID: Maybe one of these methods would be more helpful. https://bitbucket.org/mirror/cpython/src/tip/Lib/distutils/dist.py?at=default#cl-953 On Tue, Jun 25, 2013 at 12:01 PM, Yury V. Zaytsev wrote: > Hi Daniel, > > On Tue, 2013-06-25 at 08:26 -0400, Daniel Holth wrote: > >> It looks like you may be able to pass distclass=... to setup.py with a >> distutils.dist.Distribution subclass that overrides is_pure() to >> return False. > > Thank you very much for this suggestion! > > I have implemented it, but unfortunately it doesn't seem to work. I have > grepped cpython source for "is_pure" and was unable to find any > matches. > > Could you please give me some pointer as to how it is actually used? So > far, I wasn't really able to figure out where exactly in the code it is > decided whether the module goes to purelib or platlib. > > The alternative solution that I came up with is to specify the > directories in the corresponding make target: > > $(PYTHON) setup.py build --build-base=$(top_builddir)/pynest/build install --prefix=$(DESTDIR)$(prefix) --install-lib=$(DESTDIR)$(pyexecdir) --install-scripts=$(DESTDIR)$(bindir) --install-data=$(DESTDIR)$(pkgdatadir) > > It seems to work, so what do you think of this solution? > > Even considering the obvious disadvantage that setup.py called without > these parameters will not install correctly, this seems to be rather > unhorrible to me. > > -- > Sincerely yours, > Yury V. Zaytsev > > From yury at shurup.com Tue Jun 25 18:01:55 2013 From: yury at shurup.com (Yury V. Zaytsev) Date: Tue, 25 Jun 2013 18:01:55 +0200 Subject: [Distutils] Better way of forcing distutils to use platlib location? In-Reply-To: References: <1372086469.2844.49.camel@newpride> Message-ID: <1372176115.6605.56.camel@newpride> Hi Daniel, On Tue, 2013-06-25 at 08:26 -0400, Daniel Holth wrote: > It looks like you may be able to pass distclass=... to setup.py with a > distutils.dist.Distribution subclass that overrides is_pure() to > return False. Thank you very much for this suggestion! I have implemented it, but unfortunately it doesn't seem to work. I have grepped cpython source for "is_pure" and was unable to find any matches. Could you please give me some pointer as to how it is actually used? So far, I wasn't really able to figure out where exactly in the code it is decided whether the module goes to purelib or platlib. The alternative solution that I came up with is to specify the directories in the corresponding make target: $(PYTHON) setup.py build --build-base=$(top_builddir)/pynest/build install --prefix=$(DESTDIR)$(prefix) --install-lib=$(DESTDIR)$(pyexecdir) --install-scripts=$(DESTDIR)$(bindir) --install-data=$(DESTDIR)$(pkgdatadir) It seems to work, so what do you think of this solution? Even considering the obvious disadvantage that setup.py called without these parameters will not install correctly, this seems to be rather unhorrible to me. -- Sincerely yours, Yury V. Zaytsev From vinay_sajip at yahoo.co.uk Tue Jun 25 21:43:34 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 25 Jun 2013 19:43:34 +0000 (UTC) Subject: [Distutils] PEP 440 and direct references Message-ID: I would find it helpful to understand the motivation for direct references as defined in PEP 440, and some clarifications of how they can be used. For example: * Can they be used together with other clauses? As PEP 440 appears now, it seems to indicate that they can, but what does it mean in practice to have a clause which is so specific with other clauses which are less specific? * Can we have information in the "Direct reference" section summarising where such references can and can't appear? * What direct references provide which cannot be provided by the other version clauses together with (separately) URIs or paths for where archives can be found? The fact that the format of this clause is so different from all the others suggests that it doesn't really belong with them. There are also questions on other areas: * It would appear that explicit prefix matching only makes sense for == and != clauses (as it is implicit for e.g.<, > clauses). If so, can this be stated explicitly? * Since pre-/post-/dev- suffixes are ignored for the purposes of prefix matching, and given that implicit prefix matching is used with ~= , < and >, does this mean that explicit prefix matching is only useful for >=, <=? (given that == with prefix matching appears equivalent to ~=) It would also be useful to document what kinds of matches would not be possible without ~= and explicit prefix matching (trailing .*), assuming we had implicit prefix matching for ==/!= clauses. Regards, Vinay Sajip From ncoghlan at gmail.com Tue Jun 25 23:55:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Jun 2013 07:55:30 +1000 Subject: [Distutils] PEP 440 and direct references In-Reply-To: References: Message-ID: On 26 Jun 2013 05:50, "Vinay Sajip" wrote: > > I would find it helpful to understand the motivation for direct references as > defined in PEP 440, and some clarifications of how they can be used. For > example: > > * Can they be used together with other clauses? As PEP 440 appears now, it > seems to indicate that they can, but what does it mean in practice to have a > clause which is so specific with other clauses which are less specific? While the intended use case is to pin a specific version, you could also use a more general "latest" link and use other clauses to trigger an error when the version changes beyond what you expect. It's mostly because I don't think it's worth the hassle of banning such combinations, though. > * Can we have information in the "Direct reference" section summarising where > such references can and can't appear? No, because that will be up to tool developers. > * What direct references provide which cannot be provided by the other version > clauses together with (separately) URIs or paths for where archives can be > found? The fact that the format of this clause is so different from all the > others suggests that it doesn't really belong with them. I'm guided by the fact that pip and similar tools all allow direct references in requirement files. This is aimed directly at providing a standard, command line friendly notation for such references, rather than requiring the information be conveyed out of band. Most of PEP 426/440 are about the shiny, happy packaging utopia we're trying to create. This feature is aimed squarely at the messy world of the system integrator. > > There are also questions on other areas: > > * It would appear that explicit prefix matching only makes sense for == and != > clauses (as it is implicit for e.g.<, > clauses). If so, can this be stated > explicitly? It is already - none of the other clauses allow the trailing wildcard as legal syntax. > * Since pre-/post-/dev- suffixes are ignored for the purposes of prefix > matching, and given that implicit prefix matching is used with ~= , < and >, > does this mean that explicit prefix matching is only useful for >=, <=? > (given that == with prefix matching appears equivalent to ~=) Prefix matching is intended for cases where the prefix is shorter than constraints in other clauses. For example, "~= 1.3" means ">= 1.3, == 1.*", which is not the same thing as "== 1.3.*". The only time a compatible version clause and prefix matching are equivalent is for the "~= X.0" and "== X.*" case. > > It would also be useful to document what kinds of matches would not be possible > without ~= and explicit prefix matching (trailing .*), assuming we had implicit > prefix matching for ==/!= clauses. That was the approach I took in earlier versions of the spec, but people objected loudly to "==" not giving them exactly the stated version. So I switched the prefix matching to being explicit instead. The compatible release clause notation is mostly about encouraging the use of appropriate API versioning semantics (although again, with plenty of alternatives to handle the fact the real world isn't that neat and tidy). Cheers, Nick. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jun 26 00:46:54 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 25 Jun 2013 22:46:54 +0000 (UTC) Subject: [Distutils] PEP 440 and direct references References: Message-ID: Nick Coghlan gmail.com> writes: > While the intended use case is to pin a specific version, you could also > use a more general "latest" link and use other clauses to trigger an > error when the version changes beyond what you expect. > > It's mostly because I don't think it's worth the hassle of banning such > combinations, though. My question arose from thinking about how you would implement matching in cases where multiple such clauses arise. The only thing that seems to make sense is that URLs in "is" clauses must conform to the source URL format specified in PEP 426, or refer to wheels - either one allows a version number to be unambiguously derived from the URL itself. That qualification on allowable formats for URLs is not made in the "Direct Reference" section, but perhaps it should be. However, your comment above about generic "latest" links has confused me. If you can't figure out the version from the URL itself, surely making matching decisions would require downloading the URL resource and inspecting its contents? > I'm guided by the fact that pip and similar tools all allow direct references > in requirement files. This is aimed directly at providing a standard, command > line friendly notation for such references, rather than requiring the > information be conveyed out of band. Okay, but in those cases, aren't we dealing with a single clause for a particular distribution (i.e. a single "is clause" with no others), usually referring to a local "editable" project or a VCS checkout? If we have a specification "is URLA, is URLB", then unless URLA and URLB are equivalent in terms of the version they represent, both can't match, or is there some way that I've missed where they can? > Most of PEP 426/440 are about the shiny, happy packaging utopia we're trying > to create. This feature is aimed squarely at the messy world of the system > integrator. I get that, but I'm assuming that there should be support in distlib for this matching logic, and thinking about how to implement it is where I'm coming from. > It is already - none of the other clauses allow the trailing wildcard as > legal syntax. Okay, I wanted to make sure that was the intention. > Prefix matching is intended for cases where the prefix is shorter than > constraints in other clauses. For example, "~= 1.3" means ">= 1.3, == 1.*", > which is not the same thing as "== 1.3.*". OK, I see. Thanks for the answers. Regards, Vinay Sajip From ncoghlan at gmail.com Wed Jun 26 02:01:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Jun 2013 10:01:48 +1000 Subject: [Distutils] PEP 440 and direct references In-Reply-To: References: Message-ID: On 26 Jun 2013 08:52, "Vinay Sajip" wrote: > > Nick Coghlan gmail.com> writes: > > > While the intended use case is to pin a specific version, you could also > > use a more general "latest" link and use other clauses to trigger an > > error when the version changes beyond what you expect. > > > > It's mostly because I don't think it's worth the hassle of banning such > > combinations, though. > > My question arose from thinking about how you would implement matching in > cases where multiple such clauses arise. The only thing that seems to make > sense is that URLs in "is" clauses must conform to the source URL format > specified in PEP 426, or refer to wheels - either one allows a version > number to be unambiguously derived from the URL itself. That qualification > on allowable formats for URLs is not made in the "Direct Reference" section, > but perhaps it should be. However, your comment above about generic "latest" > links has confused me. If you can't figure out the version from the URL itself, > surely making matching decisions would require downloading the URL resource > and inspecting its contents? > > > I'm guided by the fact that pip and similar tools all allow direct references > > in requirement files. This is aimed directly at providing a standard, command > > line friendly notation for such references, rather than requiring the > > information be conveyed out of band. > > Okay, but in those cases, aren't we dealing with a single clause for a > particular distribution (i.e. a single "is clause" with no others), usually > referring to a local "editable" project or a VCS checkout? If we have a > specification "is URLA, is URLB", then unless URLA and URLB are equivalent in > terms of the version they represent, both can't match, or is there some way > that I've missed where they can? Hmm, you're right. So maybe it makes sense to declare that rather than be a particular kind of clause, a direct reference is instead an alternative to the entire version specifier system. That way it wouldn't allow composition with other clauses at all. More clearly constraining the URL in a direct reference to be a valid source URL, *or* have a valid wheel name as the final path component also makes a lot of sense. I'll update the PEP accordingly (not sure exactly when, though). Cheers, Nick. > > Most of PEP 426/440 are about the shiny, happy packaging utopia we're trying > > to create. This feature is aimed squarely at the messy world of the system > > integrator. > > I get that, but I'm assuming that there should be support in distlib for this > matching logic, and thinking about how to implement it is where I'm coming > from. > > > It is already - none of the other clauses allow the trailing wildcard as > > legal syntax. > > Okay, I wanted to make sure that was the intention. > > > Prefix matching is intended for cases where the prefix is shorter than > > constraints in other clauses. For example, "~= 1.3" means ">= 1.3, == 1.*", > > which is not the same thing as "== 1.3.*". > > OK, I see. Thanks for the answers. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jun 26 11:35:57 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 26 Jun 2013 09:35:57 +0000 (UTC) Subject: [Distutils] PEP 440 and direct references References: Message-ID: Nick Coghlan gmail.com> writes: > Hmm, you're right. So maybe it makes sense to declare that rather than be > a particular kind of clause, a direct reference is instead an alternative > to the entire version specifier system. That way it wouldn't allow > composition with other clauses at all. That seems fitting. Can I also suggest that "from" is better than "is", though of course it's a little longer to type? I think it reads better when it says "dist (from URL)". > I'll update the PEP accordingly (not sure exactly when, though). Okay, I'll proceed with distlib along the lines we've discussed here. Regards, Vinay Sajip From ncoghlan at gmail.com Wed Jun 26 11:41:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Jun 2013 19:41:53 +1000 Subject: [Distutils] PEP 440 and direct references In-Reply-To: References: Message-ID: On 26 Jun 2013 19:36, "Vinay Sajip" wrote: > > Nick Coghlan gmail.com> writes: > > > > Hmm, you're right. So maybe it makes sense to declare that rather than be > > a particular kind of clause, a direct reference is instead an alternative > > to the entire version specifier system. That way it wouldn't allow > > composition with other clauses at all. > > That seems fitting. Can I also suggest that "from" is better than "is", > though of course it's a little longer to type? I think it reads better when > it says "dist (from URL)". Oh, I like it. Nice :) Cheers, Nick. > > > I'll update the PEP accordingly (not sure exactly when, though). > > Okay, I'll proceed with distlib along the lines we've discussed here. > > Regards, > > Vinay Sajip > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Wed Jun 26 13:41:32 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 26 Jun 2013 14:41:32 +0300 Subject: [Distutils] Fixing PyPI download stats with real-time log analysis (Was: PyPI Download Counts) In-Reply-To: <9D98E581-B557-4478-B0D2-1D93C4060DC3@stufft.io> References: <10391719-FE9A-4FD9-B8EB-C6FE9AFA62D7@coderanger.net> <9D98E581-B557-4478-B0D2-1D93C4060DC3@stufft.io> Message-ID: On Sun, Jun 23, 2013 at 6:11 PM, Donald Stufft wrote: > > On Jun 23, 2013, at 9:36 AM, Alex Clark wrote: > > Hi Noah, > > > Noah Kantrowitz coderanger.net> writes: > > > > On Jun 22, 2013, at 10:33 PM, anatoly techtonik wrote: > > On Fri, Jun 14, 2013 at 5:05 PM, anatoly techtonik > > gmail.com> wrote: > > Could you, please, share the log format+example, so that we can > > experiment with it? > > > ping > > > Additional assistance is not required for this project. Thank you for your > > interest, and stay tuned for > > future updates. > > > > WAT. > > > Can you explain why "additional assistance is not required for this > project"? AFAICT: > > - Shit is broken (i.e. download counts) > - At first, there was no plans to fix it (which I accepted as a cost to > enable the CDN) > - Now there are plans to fix it, but no additional assistance is needed? If > that were the case I'd expect download counts to be fixed. Or at the very > least, I'd expect a "no thanks we don't need help, but here is where we are > at [ explanation of situation ]". Not "we don't need help, we'll be in > touch". I think the latter is a disservice to the community, which (I > assume) you especially don't want to be responsible for given the > tremendously positive service you've done by enabling the CDN in the first > place.) > > > We get it: download counts were not as important as CDN. But they *are* > important, so let's keep talking about how to fix them. > > > https://en.wikipedia.org/wiki/Brooks%27_law > > To further expand. Right now the biggest issue preventing the CDN download > counts is a less than ideal configuration on the VM hosting PyPI. I've been > working on (and almost have completed) a Chef cookbook that will allow us > to easily deploy PyPI to a second VM (and gracefully do the switch over) > with minimal downtime. > > However there is an issue with launching VM's at the moment that OSUOL is > looking into which means even with a completed cookbook we couldn't launch > a VM at the moment anyways. If we wanted to correct the configuration in > place it would require downtime. > I understand the pressure you're working, but I believe Brooks law doesn't apply here, because you're working on server configuration problem, while I propose to help with actual download counting problem. It is, of course, interesting to know the details of what are you trying to achieve to see where people can plug in. As I understand the problem you're trying to solve right now is (1) how to store the log or to receive it. If you're solving some problem (0) and don't have time for (1) yet - tell us the input data - so that somebody could prepare some variants. Problem (2), is to pipe the log to analyzer. I guess it will transform from "something" you got in (1) into stream of lines. Problem (3) is count processing - consume line stream, process, organize in batches and write to PyPI store. Working on (3) doesn't require any of the (0)-(2) steps to be complete. I am pushing this because I doubt that Chef configuration is an easy and fast process. At least a year ago my experience with Chef in a limited time constraint was negative (which should not come as a surprise, because I mostly see only things that are negative). I started with zero after Noah's talk at PyCon, but within 2 months I failed to provide a solution for automatic server deployment for just a couple of servers with a little bit different configuration for web and DB server than official cookbooks support. This post at http://lumberjaph.net/devops/2012/11/27/ansible-and-chef.html summarizes my experience. I am not sure you will be allowed to use Ansible for this stuff, but I'd give it a try at least for a prototype configuration. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Wed Jun 26 13:49:49 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 26 Jun 2013 07:49:49 -0400 Subject: [Distutils] Important! buildout 2.1.1 released, prepare for distribute 0.7 and buildout 2.2 Message-ID: Distribute/setuptools 0.7 bring some big changes. These changes can't be handled by buildout's normal automatic upgrade mechanism. For this reason, it's important to do one of the following if you're using buildout 2: - Pin your distribute version to <0.7dev or to some other 0.6 version. - Upgrade to buildout 2.1.1, which was just released. This will prevent automatic upgrade to distribute 0.7 or buildout 2.2. - Be prepared to re-bootstrap your buildouts with the the new buildout 2.2 bootstrap script, which will be released soon after the release of distribute 0.7. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From vinay_sajip at yahoo.co.uk Wed Jun 26 19:40:12 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 26 Jun 2013 17:40:12 +0000 (UTC) Subject: [Distutils] PEP 440 and direct references References: Message-ID: Nick Coghlan gmail.com> writes: > Hmm, you're right. So maybe it makes sense to declare that rather than be > a particular kind of clause, a direct reference is instead an alternative > to the entire version specifier system. That way it wouldn't allow > composition with other clauses at all. Another way to look at this is: the purpose of XXX in a requirement 'dist (XXX)' is to identify a specific version of "dist" to operate on. This can happen in two ways: * XXX is a direct reference - it identifies the version via URL, and no matching of any kind is required - that's the version required, plain and simple. If the resource is obtainable, then it's used, else it leads to an exception when an attempt to actually use it is made. In this case, the reason for constraining the form of the URL to a specific versioned source_url or wheel URL is twofold: (a) Repeatability (not possible with a generic "latest version" URL). (b) The ability to get the version represented by the URL, without doing any downloading, e.g. to see if a compatible version is already installed, or perhaps to see if it leads to any conflicts. * XXX is a set of version constraints which is used when querying indexes to find the most recent version which matches the constraints. According to this view, distlib does not need to provide specific support for direct references (beyond what is already provided for secure downloads, wheels etc.) - a direct reference is treated exactly the same as a single clause "(== )" for dependency resolution purposes. Of course, distlib should extend the existing version matching support to cover the additional cases in PEP 440 (such as explicit prefix matching and compatible version matching). Does that seem right? Regards, Vinay Sajip From erik.m.bray at gmail.com Wed Jun 26 22:21:55 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 26 Jun 2013 16:21:55 -0400 Subject: [Distutils] complicated setup In-Reply-To: <51BD65A5.3040500@stoneleaf.us> References: <51BD65A5.3040500@stoneleaf.us> Message-ID: On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman wrote: > Here's my file layout: > > / > |- setup.py > | > |- enum / > |- __init__.py > | > |- py2_enum.py > | > |- py3_enum.py > | > |- test / > |- test_enum.py > | > |- py2_test_enum.py > | > |- py3_test_enum.py > > __init__ and test_enum are both smart enough to pull in the correct code > when imported. The issue I am having is this: > > --8<-------------------------------------------------------------- > ethan at hydra:~$ sudo easy_install enum34 > [sudo] password for ethan: > Searching for enum34 > Reading http://pypi.python.org/simple/enum34/ > Best match: enum34 0.9 > Downloading > http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8 > Processing enum34-0.9.zip > Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg > Running enum34-0.9/setup.py -q bdist_egg --dist-dir > /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5 > SyntaxError: ('invalid syntax', > ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, ' def > __call__(cls, value, names=None, *, module=None, type=None):\n')) > > SyntaxError: ('invalid syntax', > ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, ' > class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) > > zip_safe flag not set; analyzing archive contents... > SyntaxError: ('invalid syntax', > ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py', > 211, 43, ' def __call__(cls, value, names=None, *, module=None, > type=None):\n')) > > SyntaxError: ('invalid syntax', > ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py', > 630, 47, ' class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) > > Adding enum34 0.9 to easy-install.pth file > > Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg > Processing dependencies for enum34 > Finished processing dependencies for enum34 > --8<-------------------------------------------------------------- > > distutils is trying to load the py3 versions, which of course fails on a py2 > install. The package installs successfully anyway, but if I were a user I > would be wondering if the install was trustworthy. > > It seems to me that I need to either have distutils only install the version > appropriate files, or to not try to scan the version inappropriate files, > but at this point I do not know how to do either. > > Any pointers would be greatly appreciated. That's odd. I work on a package that ships Python 2 and Python 3 versions of some modules and I have never seen this problem before. Perhaps you could post your setup.py? Erik From donald at stufft.io Wed Jun 26 22:23:08 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 26 Jun 2013 16:23:08 -0400 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> Message-ID: On Jun 26, 2013, at 4:21 PM, Erik Bray wrote: > On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman wrote: >> Here's my file layout: >> >> / >> |- setup.py >> | >> |- enum / >> |- __init__.py >> | >> |- py2_enum.py >> | >> |- py3_enum.py >> | >> |- test / >> |- test_enum.py >> | >> |- py2_test_enum.py >> | >> |- py3_test_enum.py >> >> __init__ and test_enum are both smart enough to pull in the correct code >> when imported. The issue I am having is this: >> >> --8<-------------------------------------------------------------- >> ethan at hydra:~$ sudo easy_install enum34 >> [sudo] password for ethan: >> Searching for enum34 >> Reading http://pypi.python.org/simple/enum34/ >> Best match: enum34 0.9 >> Downloading >> http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8 >> Processing enum34-0.9.zip >> Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg >> Running enum34-0.9/setup.py -q bdist_egg --dist-dir >> /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5 >> SyntaxError: ('invalid syntax', >> ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, ' def >> __call__(cls, value, names=None, *, module=None, type=None):\n')) >> >> SyntaxError: ('invalid syntax', >> ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, ' >> class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >> >> zip_safe flag not set; analyzing archive contents... >> SyntaxError: ('invalid syntax', >> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py', >> 211, 43, ' def __call__(cls, value, names=None, *, module=None, >> type=None):\n')) >> >> SyntaxError: ('invalid syntax', >> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py', >> 630, 47, ' class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >> >> Adding enum34 0.9 to easy-install.pth file >> >> Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg >> Processing dependencies for enum34 >> Finished processing dependencies for enum34 >> --8<-------------------------------------------------------------- >> >> distutils is trying to load the py3 versions, which of course fails on a py2 >> install. The package installs successfully anyway, but if I were a user I >> would be wondering if the install was trustworthy. >> >> It seems to me that I need to either have distutils only install the version >> appropriate files, or to not try to scan the version inappropriate files, >> but at this point I do not know how to do either. >> >> Any pointers would be greatly appreciated. > > That's odd. I work on a package that ships Python 2 and Python 3 > versions of some modules and I have never seen this problem before. > Perhaps you could post your setup.py? > > Erik > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig If I recall this is because it's trying to compile byte code. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From carl at oddbird.net Wed Jun 26 22:35:24 2013 From: carl at oddbird.net (Carl Meyer) Date: Wed, 26 Jun 2013 14:35:24 -0600 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> Message-ID: <51CB508C.70307@oddbird.net> On 06/26/2013 02:23 PM, Donald Stufft wrote: > > On Jun 26, 2013, at 4:21 PM, Erik Bray wrote: > >> On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman wrote: >>> Here's my file layout: >>> >>> / >>> |- setup.py >>> | >>> |- enum / >>> |- __init__.py >>> | >>> |- py2_enum.py >>> | >>> |- py3_enum.py >>> | >>> |- test / >>> |- test_enum.py >>> | >>> |- py2_test_enum.py >>> | >>> |- py3_test_enum.py >>> >>> __init__ and test_enum are both smart enough to pull in the correct code >>> when imported. The issue I am having is this: >>> >>> --8<-------------------------------------------------------------- >>> ethan at hydra:~$ sudo easy_install enum34 >>> [sudo] password for ethan: >>> Searching for enum34 >>> Reading http://pypi.python.org/simple/enum34/ >>> Best match: enum34 0.9 >>> Downloading >>> http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8 >>> Processing enum34-0.9.zip >>> Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg >>> Running enum34-0.9/setup.py -q bdist_egg --dist-dir >>> /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5 >>> SyntaxError: ('invalid syntax', >>> ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, ' def >>> __call__(cls, value, names=None, *, module=None, type=None):\n')) >>> >>> SyntaxError: ('invalid syntax', >>> ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, ' >>> class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >>> >>> zip_safe flag not set; analyzing archive contents... >>> SyntaxError: ('invalid syntax', >>> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py', >>> 211, 43, ' def __call__(cls, value, names=None, *, module=None, >>> type=None):\n')) >>> >>> SyntaxError: ('invalid syntax', >>> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py', >>> 630, 47, ' class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >>> >>> Adding enum34 0.9 to easy-install.pth file >>> >>> Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg >>> Processing dependencies for enum34 >>> Finished processing dependencies for enum34 >>> --8<-------------------------------------------------------------- >>> >>> distutils is trying to load the py3 versions, which of course fails on a py2 >>> install. The package installs successfully anyway, but if I were a user I >>> would be wondering if the install was trustworthy. >>> >>> It seems to me that I need to either have distutils only install the version >>> appropriate files, or to not try to scan the version inappropriate files, >>> but at this point I do not know how to do either. >>> >>> Any pointers would be greatly appreciated. >> >> That's odd. I work on a package that ships Python 2 and Python 3 >> versions of some modules and I have never seen this problem before. >> Perhaps you could post your setup.py? > > If I recall this is because it's trying to compile byte code. That's correct. And you'll note that despite the scary-looking syntax errors from byte-compilation, the install actually succeeded. Carl From ethan at stoneleaf.us Wed Jun 26 23:24:20 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Jun 2013 14:24:20 -0700 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> Message-ID: <51CB5C04.3080806@stoneleaf.us> On 06/26/2013 01:21 PM, Erik Bray wrote: > On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman wrote: >> Here's my file layout: >> >> / >> |- setup.py >> | >> |- enum / >> |- __init__.py >> | >> |- py2_enum.py >> | >> |- py3_enum.py >> | >> |- test / >> |- test_enum.py >> | >> |- py2_test_enum.py >> | >> |- py3_test_enum.py > > That's odd. I work on a package that ships Python 2 and Python 3 > versions of some modules and I have never seen this problem before. > Perhaps you could post your setup.py? --8< -- setup.py ----------------------------------------------------------------- from distutils.core import setup long_desc = open('enum/enum.rst').read() setup( name='enum34', version='0.9', url='https://pypi.python.org/pypi/enum34', packages=['enum'], package_data={ 'enum':[ 'enum.rst', 'enum.pdf', 'test/test_enum.py', 'test/py2_test_enum.py', 'test/py3_test_enum.py', ], }, license='BSD License', description='Python 3.4 Enum backported to 3.3, 3.2, 3.1, 2.7, 2.6, 2.5, and 2.4', long_description=long_desc, provides=['enum'], author='Ethan Furman', author_email='ethan at stoneleaf.us', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'License :: OSI Approved :: BSD License', 'Programming Language :: Python', 'Topic :: Software Development' ], ) --8< -- setup.py ----------------------------------------------------------------- -- ~Ethan~ From ethan at stoneleaf.us Wed Jun 26 23:19:53 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Jun 2013 14:19:53 -0700 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> Message-ID: <51CB5AF9.5040807@stoneleaf.us> On 06/26/2013 01:23 PM, Donald Stufft wrote: > > If I recall this is because it's trying to compile byte code. Is there an option to tell distutils not to compile byte code? -- ~Ethan~ From ethan at stoneleaf.us Wed Jun 26 23:37:38 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Jun 2013 14:37:38 -0700 Subject: [Distutils] complicated setup In-Reply-To: <51CB508C.70307@oddbird.net> References: <51BD65A5.3040500@stoneleaf.us> <51CB508C.70307@oddbird.net> Message-ID: <51CB5F22.40307@stoneleaf.us> On 06/26/2013 01:35 PM, Carl Meyer wrote: > On 06/26/2013 02:23 PM, Donald Stufft wrote: >> >> If I recall this is because it's trying to compile byte code. > > That's correct. And you'll note that despite the scary-looking syntax > errors from byte-compilation, the install actually succeeded. I did notice that the install was successful. I'm mostly concerned for users getting the scary messages. -- ~Ethan~ From pje at telecommunity.com Thu Jun 27 05:12:46 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 26 Jun 2013 23:12:46 -0400 Subject: [Distutils] PEP 440 and direct references In-Reply-To: References: Message-ID: On Wed, Jun 26, 2013 at 1:40 PM, Vinay Sajip wrote: > (a) Repeatability (not possible with a generic "latest version" URL). ISTM that forcing repeatability in the spec implies the inability to do development with requirements that are also in development, unless there is also an out-of-band channel for communicating the URL. From ncoghlan at gmail.com Thu Jun 27 07:36:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Jun 2013 15:36:59 +1000 Subject: [Distutils] PEP 440 and direct references In-Reply-To: References: Message-ID: On 27 June 2013 13:12, PJ Eby wrote: > On Wed, Jun 26, 2013 at 1:40 PM, Vinay Sajip wrote: >> (a) Repeatability (not possible with a generic "latest version" URL). > > ISTM that forcing repeatability in the spec implies the inability to > do development with requirements that are also in development, unless > there is also an out-of-band channel for communicating the URL. Yeah, I think you might be right here. Specific tools that want to ensure repeatability are free to impose restrictions like that (just as they're free to rule out the use of direct references entirely), but we don't want to constraint them too much at the specification level or they lose their effectiveness as an escape clause. The description already says "The exact URLs and targets supported will be tool dependent." I think it's OK for distlib to require that by the time any associated info is handed over as a requirements specification, it has been converted to a specific version number. It isn't distlib's problem whether that was extracted from the URL directly, or by downloading it and looking at (or possibly even generating) the metadata. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Thu Jun 27 11:16:04 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 27 Jun 2013 09:16:04 +0000 (UTC) Subject: [Distutils] PEP 440 and prefix matching Message-ID: All the examples of explicit prefix matching in PEP 440 show only release clause components, but it's not explicitly stated whether that's specifically intended. For example, are prefix matching clauses such as == 1.2.dev0.* != 1.2.post1.* allowed? Regards, Vinay Sajip From peter at sabaini.at Thu Jun 27 10:36:23 2013 From: peter at sabaini.at (Peter Sabaini) Date: Thu, 27 Jun 2013 10:36:23 +0200 Subject: [Distutils] setuptools 0.8 on pypi? Message-ID: <51CBF987.8070404@sabaini.at> Installation instructions for setuptools, eg. here[1], mention setuptools 0.8 as a download from pypi. However, pypi doesn't seem to carry version 0.8, only up to 0.7.4[2]. Are the installation instructions still work-in-progress, and is 0.7.4 the recommended version ATM? Thanks for the clarification, peter. [1] https://bitbucket.org/pypa/setuptools [2] https://pypi.python.org/pypi/setuptools From ncoghlan at gmail.com Thu Jun 27 13:24:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Jun 2013 21:24:06 +1000 Subject: [Distutils] PEP 440 and prefix matching In-Reply-To: References: Message-ID: On 27 June 2013 19:16, Vinay Sajip wrote: > All the examples of explicit prefix matching in PEP 440 show only release > clause components, but it's not explicitly stated whether that's > specifically intended. For example, are prefix matching clauses such as > > == 1.2.dev0.* > != 1.2.post1.* > > allowed? Sure. I'm not sure why anyone would actually want to do it, but it's allowed (disallowing it would actually make the spec more complicated rather than simpler). The only thing which explicitly ignores the suffix is the compatible release clause: "If a pre-release, post-release or developmental release is named in a compatible release clause as V.N.suffix, then the suffix is ignored when determining the required prefix match" The rationale for that restriction is that we're mostly using the same compatible release semantics as semantic versioning, except that we assume a reasonable level of backwards compatibility will exist even for 0.x releases, leaving it to people to use prefix matching if a dependency exposes a somewhat unstable API. Semver doesn't define compatibility semantics outside the release number, so we don't either. It occurs to me, though, given that we now have an exact prefix matching notation for more unusual forward compatibility constraints, we *could* just make all compatible release clauses explicitly semantic versioning based. Then you could say things like (2.2.2) or (~= 2.2.2) to mean (>= 2.2.2, == 2.*). Under the current PEP, those clauses would translate to (>= 2.2.2, == 2.2.*), which differs for no good reason I can see from the handling of a clause like (~= 2.2.post1), which would translate as (>= 2.2.post1, == 2.*). I think the semantic versioning based system is actually easier to explain than the current scheme (mostly because it's more consistent), so unless someone comes up with a compelling counterargument in the next few days, I'll switch the specification to always use "== V.*" as the prefix matching component of a compatible release clause, no *matter* how many components there are in the version specifier itself. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Thu Jun 27 14:45:34 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 27 Jun 2013 08:45:34 -0400 Subject: [Distutils] PEP 440 and prefix matching In-Reply-To: References: Message-ID: FYI semver 2.0.0 final was released :-) semver.org On Thu, Jun 27, 2013 at 7:24 AM, Nick Coghlan wrote: > On 27 June 2013 19:16, Vinay Sajip wrote: >> All the examples of explicit prefix matching in PEP 440 show only release >> clause components, but it's not explicitly stated whether that's >> specifically intended. For example, are prefix matching clauses such as >> >> == 1.2.dev0.* >> != 1.2.post1.* >> >> allowed? > > Sure. I'm not sure why anyone would actually want to do it, but it's > allowed (disallowing it would actually make the spec more complicated > rather than simpler). > > The only thing which explicitly ignores the suffix is the compatible > release clause: "If a pre-release, post-release or developmental > release is named in a compatible release clause as V.N.suffix, then > the suffix is ignored when determining the required prefix match" > > The rationale for that restriction is that we're mostly using the same > compatible release semantics as semantic versioning, except that we > assume a reasonable level of backwards compatibility will exist even > for 0.x releases, leaving it to people to use prefix matching if a > dependency exposes a somewhat unstable API. Semver doesn't define > compatibility semantics outside the release number, so we don't > either. > > It occurs to me, though, given that we now have an exact prefix > matching notation for more unusual forward compatibility constraints, > we *could* just make all compatible release clauses explicitly > semantic versioning based. Then you could say things like (2.2.2) or > (~= 2.2.2) to mean (>= 2.2.2, == 2.*). Under the current PEP, those > clauses would translate to (>= 2.2.2, == 2.2.*), which differs for no > good reason I can see from the handling of a clause like (~= > 2.2.post1), which would translate as (>= 2.2.post1, == 2.*). > > I think the semantic versioning based system is actually easier to > explain than the current scheme (mostly because it's more consistent), > so unless someone comes up with a compelling counterargument in the > next few days, I'll switch the specification to always use "== V.*" as > the prefix matching component of a compatible release clause, no > *matter* how many components there are in the version specifier > itself. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Thu Jun 27 16:15:41 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 27 Jun 2013 10:15:41 -0400 Subject: [Distutils] PEP 440 and prefix matching In-Reply-To: References: Message-ID: <366B2BBE-D638-41D0-B0A7-10E33A736108@stufft.io> On Jun 27, 2013, at 7:24 AM, Nick Coghlan wrote: > It occurs to me, though, given that we now have an exact prefix > matching notation for more unusual forward compatibility constraints, > we *could* just make all compatible release clauses explicitly > semantic versioning based. Then you could say things like (2.2.2) or > (~= 2.2.2) to mean (>= 2.2.2, == 2.*). Under the current PEP, those > clauses would translate to (>= 2.2.2, == 2.2.*), which differs for no > good reason I can see from the handling of a clause like (~= > 2.2.post1), which would translate as (>= 2.2.post1, == 2.*). > > I think the semantic versioning based system is actually easier to > explain than the current scheme (mostly because it's more consistent), > so unless someone comes up with a compelling counterargument in the > next few days, I'll switch the specification to always use "== V.*" as > the prefix matching component of a compatible release clause, no > *matter* how many components there are in the version specifier > itself. No don't do that. Not everything follows semver like that. For instance Django often times needs changes between a 1.N and a 1.N+1, but 1.N.* is supposed to be as backwards compatible as possible to make it simple for people to get security updates. Forcing it to always be N.* feels way to opinionated for something like versioning across all of the Python ecosystem. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Thu Jun 27 16:43:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Jun 2013 00:43:15 +1000 Subject: [Distutils] PEP 440 and prefix matching In-Reply-To: <366B2BBE-D638-41D0-B0A7-10E33A736108@stufft.io> References: <366B2BBE-D638-41D0-B0A7-10E33A736108@stufft.io> Message-ID: On 28 June 2013 00:15, Donald Stufft wrote: > > On Jun 27, 2013, at 7:24 AM, Nick Coghlan wrote: >> It occurs to me, though, given that we now have an exact prefix >> matching notation for more unusual forward compatibility constraints, >> we *could* just make all compatible release clauses explicitly >> semantic versioning based. Then you could say things like (2.2.2) or >> (~= 2.2.2) to mean (>= 2.2.2, == 2.*). Under the current PEP, those >> clauses would translate to (>= 2.2.2, == 2.2.*), which differs for no >> good reason I can see from the handling of a clause like (~= >> 2.2.post1), which would translate as (>= 2.2.post1, == 2.*). >> >> I think the semantic versioning based system is actually easier to >> explain than the current scheme (mostly because it's more consistent), >> so unless someone comes up with a compelling counterargument in the >> next few days, I'll switch the specification to always use "== V.*" as >> the prefix matching component of a compatible release clause, no >> *matter* how many components there are in the version specifier >> itself. > > No don't do that. Not everything follows semver like that. For instance Django > often times needs changes between a 1.N and a 1.N+1, but 1.N.* is supposed > to be as backwards compatible as possible to make it simple for people to > get security updates. Forcing it to always be N.* feels way to opinionated for > something like versioning across all of the Python ecosystem. I think that counts as a compelling counterargument - I guess I'm leaving the PEP alone on that front, then :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Thu Jun 27 17:11:42 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 27 Jun 2013 17:11:42 +0200 Subject: [Distutils] setuptools 0.8 on pypi? In-Reply-To: <51CBF987.8070404@sabaini.at> References: <51CBF987.8070404@sabaini.at> Message-ID: On Thu, Jun 27, 2013 at 10:36 AM, Peter Sabaini wrote: > [1] https://bitbucket.org/pypa/setuptools This is the source repository for setuptools. What you are seeing is the version in development, which is 0.8. It's not released yet. The pypi page contains the updated readme, and points to the official documentation: http://pythonhosted.org/setuptools/ //Lennart From erik.m.bray at gmail.com Thu Jun 27 19:19:14 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Thu, 27 Jun 2013 13:19:14 -0400 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> Message-ID: On Wed, Jun 26, 2013 at 4:21 PM, Erik Bray wrote: > On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman wrote: >> Here's my file layout: >> >> / >> |- setup.py >> | >> |- enum / >> |- __init__.py >> | >> |- py2_enum.py >> | >> |- py3_enum.py >> | >> |- test / >> |- test_enum.py >> | >> |- py2_test_enum.py >> | >> |- py3_test_enum.py >> >> __init__ and test_enum are both smart enough to pull in the correct code >> when imported. The issue I am having is this: >> >> --8<-------------------------------------------------------------- >> ethan at hydra:~$ sudo easy_install enum34 >> [sudo] password for ethan: >> Searching for enum34 >> Reading http://pypi.python.org/simple/enum34/ >> Best match: enum34 0.9 >> Downloading >> http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8 >> Processing enum34-0.9.zip >> Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg >> Running enum34-0.9/setup.py -q bdist_egg --dist-dir >> /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5 >> SyntaxError: ('invalid syntax', >> ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, ' def >> __call__(cls, value, names=None, *, module=None, type=None):\n')) >> >> SyntaxError: ('invalid syntax', >> ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, ' >> class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >> >> zip_safe flag not set; analyzing archive contents... >> SyntaxError: ('invalid syntax', >> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py', >> 211, 43, ' def __call__(cls, value, names=None, *, module=None, >> type=None):\n')) >> >> SyntaxError: ('invalid syntax', >> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py', >> 630, 47, ' class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >> >> Adding enum34 0.9 to easy-install.pth file >> >> Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg >> Processing dependencies for enum34 >> Finished processing dependencies for enum34 >> --8<-------------------------------------------------------------- >> >> distutils is trying to load the py3 versions, which of course fails on a py2 >> install. The package installs successfully anyway, but if I were a user I >> would be wondering if the install was trustworthy. >> >> It seems to me that I need to either have distutils only install the version >> appropriate files, or to not try to scan the version inappropriate files, >> but at this point I do not know how to do either. >> >> Any pointers would be greatly appreciated. > > That's odd. I work on a package that ships Python 2 and Python 3 > versions of some modules and I have never seen this problem before. > Perhaps you could post your setup.py? It occur to me now that the reason I don't see this kind of error on my project is because in the setup.py I am just excluding the version-specific files based on what Python version the user has. Perhaps you should do the same--only install the Python 2 tests on Python 2 and so on. Just make sure that the MANIFEST.in is set up to include both versions in the source distribution. As long as you don't install the Py2 files in Py3 or vice versa it shoudn't try to compile the bytecode for those files. Erik From peter at sabaini.at Thu Jun 27 17:20:44 2013 From: peter at sabaini.at (Peter Sabaini) Date: Thu, 27 Jun 2013 17:20:44 +0200 Subject: [Distutils] setuptools 0.8 on pypi? In-Reply-To: References: <51CBF987.8070404@sabaini.at> Message-ID: <51CC584C.7050108@sabaini.at> On 27.06.13 17:11, Lennart Regebro wrote: > On Thu, Jun 27, 2013 at 10:36 AM, Peter Sabaini wrote: >> [1] https://bitbucket.org/pypa/setuptools > > This is the source repository for setuptools. What you are seeing is > the version in development, which is 0.8. It's not released yet. > > The pypi page contains the updated readme, and points to the official > documentation: Oh, I see... I guess I mistook it for a released version because http://www.pip-installer.org references it, and https://pypi.python.org/pypi/setuptools doesn't really say much :-) Thanks for the clarification. peter. > > http://pythonhosted.org/setuptools/ > > //Lennart > From regebro at gmail.com Thu Jun 27 19:28:08 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 27 Jun 2013 19:28:08 +0200 Subject: [Distutils] setuptools 0.8 on pypi? In-Reply-To: <51CC584C.7050108@sabaini.at> References: <51CBF987.8070404@sabaini.at> <51CC584C.7050108@sabaini.at> Message-ID: On Thu, Jun 27, 2013 at 5:20 PM, Peter Sabaini wrote: > Oh, I see... I guess I mistook it for a released version because > http://www.pip-installer.org references it Aha. I'd guess that link would better go to the documentation. From ethan at stoneleaf.us Thu Jun 27 19:31:33 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Jun 2013 10:31:33 -0700 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> Message-ID: <51CC76F5.3030208@stoneleaf.us> On 06/27/2013 10:19 AM, Erik Bray wrote: > On Wed, Jun 26, 2013 at 4:21 PM, Erik Bray wrote: >> On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman wrote: >>> Here's my file layout: >>> >>> / >>> |- setup.py >>> | >>> |- enum / >>> |- __init__.py >>> | >>> |- py2_enum.py >>> | >>> |- py3_enum.py >>> | >>> |- test / >>> |- test_enum.py >>> | >>> |- py2_test_enum.py >>> | >>> |- py3_test_enum.py >>> >>> __init__ and test_enum are both smart enough to pull in the correct code >>> when imported. The issue I am having is this: >>> >>> --8<-------------------------------------------------------------- >>> ethan at hydra:~$ sudo easy_install enum34 >>> [sudo] password for ethan: >>> Searching for enum34 >>> Reading http://pypi.python.org/simple/enum34/ >>> Best match: enum34 0.9 >>> Downloading >>> http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8 >>> Processing enum34-0.9.zip >>> Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg >>> Running enum34-0.9/setup.py -q bdist_egg --dist-dir >>> /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5 >>> SyntaxError: ('invalid syntax', >>> ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, ' def >>> __call__(cls, value, names=None, *, module=None, type=None):\n')) >>> >>> SyntaxError: ('invalid syntax', >>> ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, ' >>> class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >>> >>> zip_safe flag not set; analyzing archive contents... >>> SyntaxError: ('invalid syntax', >>> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py', >>> 211, 43, ' def __call__(cls, value, names=None, *, module=None, >>> type=None):\n')) >>> >>> SyntaxError: ('invalid syntax', >>> ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py', >>> 630, 47, ' class AutoNumberedEnum(Enum, metaclass=auto_enum):\n')) >>> >>> Adding enum34 0.9 to easy-install.pth file >>> >>> Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg >>> Processing dependencies for enum34 >>> Finished processing dependencies for enum34 >>> --8<-------------------------------------------------------------- >>> >>> distutils is trying to load the py3 versions, which of course fails on a py2 >>> install. The package installs successfully anyway, but if I were a user I >>> would be wondering if the install was trustworthy. >>> >>> It seems to me that I need to either have distutils only install the version >>> appropriate files, or to not try to scan the version inappropriate files, >>> but at this point I do not know how to do either. >>> >>> Any pointers would be greatly appreciated. >> >> That's odd. I work on a package that ships Python 2 and Python 3 >> versions of some modules and I have never seen this problem before. >> Perhaps you could post your setup.py? > > It occur to me now that the reason I don't see this kind of error on > my project is because in the setup.py I am just excluding the > version-specific files based on what Python version the user has. > Perhaps you should do the same--only install the Python 2 tests on > Python 2 and so on. Just make sure that the MANIFEST.in is set up to > include both versions in the source distribution. As long as you > don't install the Py2 files in Py3 or vice versa it shoudn't try to > compile the bytecode for those files. I would be willing to do that, but I don't know how, and so far my searching hasn't yielded anything useful besides this mailing list. -- ~Ethan~ From merwok at netwok.org Thu Jun 27 19:59:30 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 27 Jun 2013 13:59:30 -0400 Subject: [Distutils] complicated setup In-Reply-To: <51CB5AF9.5040807@stoneleaf.us> References: <51BD65A5.3040500@stoneleaf.us> <51CB5AF9.5040807@stoneleaf.us> Message-ID: <51CC7D82.10405@netwok.org> Le 26/06/2013 17:19, Ethan Furman a ?crit : > Is there an option to tell distutils not to compile byte code? Yes. setup.py build --help should show you the options, otherwise see http://hg.python.org/cpython/file/dad02a080bbc/Doc/packaging/commandref.rst#l113 (I should backport that doc change to distutils). Regards From oscar.j.benjamin at gmail.com Thu Jun 27 20:55:11 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 27 Jun 2013 19:55:11 +0100 Subject: [Distutils] complicated setup In-Reply-To: <51CC76F5.3030208@stoneleaf.us> References: <51BD65A5.3040500@stoneleaf.us> <51CC76F5.3030208@stoneleaf.us> Message-ID: On 27 June 2013 18:31, Ethan Furman wrote: >> >> It occur to me now that the reason I don't see this kind of error on >> my project is because in the setup.py I am just excluding the >> version-specific files based on what Python version the user has. >> Perhaps you should do the same--only install the Python 2 tests on >> Python 2 and so on. Just make sure that the MANIFEST.in is set up to >> include both versions in the source distribution. As long as you >> don't install the Py2 files in Py3 or vice versa it shoudn't try to >> compile the bytecode for those files. > > I would be willing to do that, but I don't know how, and so far my searching > hasn't yielded anything useful besides this mailing list. Do this do what you want: #setup.py import sys from distutils.core import setup package_data = {'enum': [...]} if sys.version_info >= (3,0): package_data['enum'].append( 'test/py3_test_enum.py') else: package_data['enum'].append( 'test/py2_test_enum.py') setup(package_data = package_data, ...) Oscar From donald at stufft.io Thu Jun 27 23:18:22 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 27 Jun 2013 17:18:22 -0400 Subject: [Distutils] Download Counts on PyPI Message-ID: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> Download counts have (kind of) been re-enabled on PyPI. The new download counts work as so: * At the bottom of a detail page there is rolling tallies for the last day, the last week, and the last month[1] * The API exposes total download counts per file still. * The rolling counts update more or less in real time while the total counts in the API update hourly. * We no longer incorporate download counts from mirrors, making the mirroring infrastructure a tree and not a federation. [1] Only the last day is enabled currently, once more data is available the last week and last month will also be enabled. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Fri Jun 28 05:24:35 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 27 Jun 2013 23:24:35 -0400 Subject: [Distutils] Download Counts on PyPI In-Reply-To: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> References: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> Message-ID: On Thu, Jun 27, 2013 at 5:18 PM, Donald Stufft wrote: > Download counts have (kind of) been re-enabled on PyPI. > > The new download counts work as so: > > * At the bottom of a detail page there is rolling tallies for the last > day, the last week, and the last month[1] > * The API exposes total download counts per file still. > * The rolling counts update more or less in real time while the total counts in the API update hourly. > * We no longer incorporate download counts from mirrors, making the mirroring infrastructure a tree and not a federation. > Is there any plan for per-version/per-file stats? I notice that the current stat display is across all versions, which makes it harder to tell e.g. setuptools 0.7 uptake vs 0.6. From donald at stufft.io Fri Jun 28 05:44:14 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 27 Jun 2013 23:44:14 -0400 Subject: [Distutils] Download Counts on PyPI In-Reply-To: References: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> Message-ID: <27DAC8B7-4860-4984-870E-DEF8480CBDBF@stufft.io> On Jun 27, 2013, at 11:24 PM, PJ Eby wrote: > On Thu, Jun 27, 2013 at 5:18 PM, Donald Stufft wrote: >> Download counts have (kind of) been re-enabled on PyPI. >> >> The new download counts work as so: >> >> * At the bottom of a detail page there is rolling tallies for the last >> day, the last week, and the last month[1] >> * The API exposes total download counts per file still. >> * The rolling counts update more or less in real time while the total counts in the API update hourly. >> * We no longer incorporate download counts from mirrors, making the mirroring infrastructure a tree and not a federation. >> > > Is there any plan for per-version/per-file stats? I notice that the > current stat display is across all versions, which makes it harder to > tell e.g. setuptools 0.7 uptake vs 0.6. That information is available via the API in a total format (basically the old download formats). I think for someone evaluating the popularity of a package the new format is much more useful. Obviously there is more information that _could_ be displayed (with varying amounts of effort) but you run the risk of overwhelming the user with stats. There's a lot more I'd like to do in general statistics wise geared towards different targeted audiences. There aren't exact plans right now as there are more pressing concerns and it needs to be done in a way that tells a useful "story" without overwhelming people with numbers. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From aclark at aclark.net Fri Jun 28 13:36:19 2013 From: aclark at aclark.net (Alex Clark) Date: Fri, 28 Jun 2013 11:36:19 +0000 (UTC) Subject: [Distutils] Download Counts on PyPI References: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> <27DAC8B7-4860-4984-870E-DEF8480CBDBF@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > > Is there any plan for per-version/per-file stats? I notice that the > > current stat display is across all versions, which makes it harder to > > tell e.g. setuptools 0.7 uptake vs 0.6. > > That information is available via the API in a total format (basically the old download formats). > > I think for someone evaluating the popularity of a package the new format is much more useful. Obviously > there is more information that _could_ be displayed (with varying amounts of effort) but you run the risk > of overwhelming the user with stats. > > There's a lot more I'd like to do in general statistics wise geared towards different targeted audiences. > There aren't exact plans right now as there are more pressing concerns and it needs to be done in a way that > tells a useful "story" without overwhelming people with numbers. Are you sure it doesn't "just work"? Vanity seems to work as expected now, I just cut a new release: aclark at Alexs-MacBook-Pro:~/ > vanity vanity vanity-1.0.zip 2011-04-14 980 vanity-1.1.0.zip 2011-10-26 771 vanity-1.1.1.zip 2011-10-28 744 vanity-1.1.2.zip 2011-10-28 775 vanity-1.2.0.tar.gz 2012-01-30 712 vanity-1.2.1.tar.gz 2012-02-16 761 vanity-1.2.2.zip 2012-07-31 474 vanity-1.2.3.zip 2012-08-08 523 vanity-1.2.4.zip 2013-02-19 249 vanity-1.2.5.zip 2013-03-18 252 vanity-2.0.0.zip 2013-05-26 3 vanity-2.0.1.zip 2013-05-27 0 vanity-2.0.2.zip 2013-05-27 2 vanity-2.0.3.zip 2013-06-28 0 ---------------------------------------------- vanity has been downloaded 6,246 times! aclark at Alexs-MacBook-Pro:~/ > vanity vanity==2.0.2 vanity-2.0.2.zip 2013-05-27 2 ------------------------------------------- vanity 2.0.2 has been downloaded 2 times! I don't necessarily care what is displayed on PyPI, as long as we have (roughly) the same functionality as before. Thanks for working on this! From donald at stufft.io Fri Jun 28 17:52:47 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 28 Jun 2013 11:52:47 -0400 Subject: [Distutils] Download Counts on PyPI In-Reply-To: References: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> <27DAC8B7-4860-4984-870E-DEF8480CBDBF@stufft.io> Message-ID: <1345A8BE-C1E1-4A67-A85F-FC3B25D1BC79@stufft.io> On Jun 28, 2013, at 7:36 AM, Alex Clark wrote: > Donald Stufft stufft.io> writes: > > >>> Is there any plan for per-version/per-file stats? I notice that the >>> current stat display is across all versions, which makes it harder to >>> tell e.g. setuptools 0.7 uptake vs 0.6. >> >> That information is available via the API in a total format (basically the > old download formats). >> >> I think for someone evaluating the popularity of a package the new format > is much more useful. Obviously >> there is more information that _could_ be displayed (with varying amounts > of effort) but you run the risk >> of overwhelming the user with stats. >> >> There's a lot more I'd like to do in general statistics wise geared > towards different targeted audiences. >> There aren't exact plans right now as there are more pressing concerns and > it needs to be done in a way that >> tells a useful "story" without overwhelming people with numbers. > > > Are you sure it doesn't "just work"? Vanity seems to work as expected now, I > just cut a new release: Yes the API contains the old numbers. (That is, they are updating, but it is by file totals). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Fri Jun 28 23:39:48 2013 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 28 Jun 2013 14:39:48 -0700 Subject: [Distutils] setuptools 0.8 on pypi? In-Reply-To: <51CC584C.7050108@sabaini.at> References: <51CBF987.8070404@sabaini.at> <51CC584C.7050108@sabaini.at> Message-ID: > Oh, I see... I guess I mistook it for a released version because > http://www.pip-installer.org references it, woops, those are the 1.4dev docs (unreleased). I'll add a note that setuptools-0.8 is still not released Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Jun 29 02:09:03 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 28 Jun 2013 17:09:03 -0700 Subject: [Distutils] complicated setup In-Reply-To: <51CE201B.7070509@stoneleaf.us> References: <51BD65A5.3040500@stoneleaf.us> <51CC76F5.3030208@stoneleaf.us> <51CE201B.7070509@stoneleaf.us> Message-ID: <51CE259F.30000@stoneleaf.us> On 06/28/2013 04:45 PM, Ethan Furman wrote: > On 06/27/2013 11:55 AM, Oscar Benjamin wrote: >> On 27 June 2013 18:31, Ethan Furman wrote: >>>> >>>> It occur to me now that the reason I don't see this kind of error on >>>> my project is because in the setup.py I am just excluding the >>>> version-specific files based on what Python version the user has. >>>> Perhaps you should do the same--only install the Python 2 tests on >>>> Python 2 and so on. Just make sure that the MANIFEST.in is set up to >>>> include both versions in the source distribution. As long as you >>>> don't install the Py2 files in Py3 or vice versa it shoudn't try to >>>> compile the bytecode for those files. >>> >>> I would be willing to do that, but I don't know how, and so far my searching >>> hasn't yielded anything useful besides this mailing list. >> >> Do this do what you want: >> >> #setup.py >> import sys >> from distutils.core import setup >> >> package_data = {'enum': [...]} >> >> if sys.version_info >= (3,0): >> package_data['enum'].append( 'test/py3_test_enum.py') >> else: >> package_data['enum'].append( 'test/py2_test_enum.py') >> >> setup(package_data = package_data, ...) > > Cool! > > Is there a way to have files renamed? Ideally the installed package would just have enum.py, test_enum.py, and not > py2_enum.py and py2_test_enum.py Am I making this too complicated? Can I just do my renaming in the setup.py script before calling the setup function? -- ~Ethan~ From ethan at stoneleaf.us Sat Jun 29 01:45:31 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 28 Jun 2013 16:45:31 -0700 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> <51CC76F5.3030208@stoneleaf.us> Message-ID: <51CE201B.7070509@stoneleaf.us> On 06/27/2013 11:55 AM, Oscar Benjamin wrote: > On 27 June 2013 18:31, Ethan Furman wrote: >>> >>> It occur to me now that the reason I don't see this kind of error on >>> my project is because in the setup.py I am just excluding the >>> version-specific files based on what Python version the user has. >>> Perhaps you should do the same--only install the Python 2 tests on >>> Python 2 and so on. Just make sure that the MANIFEST.in is set up to >>> include both versions in the source distribution. As long as you >>> don't install the Py2 files in Py3 or vice versa it shoudn't try to >>> compile the bytecode for those files. >> >> I would be willing to do that, but I don't know how, and so far my searching >> hasn't yielded anything useful besides this mailing list. > > Do this do what you want: > > #setup.py > import sys > from distutils.core import setup > > package_data = {'enum': [...]} > > if sys.version_info >= (3,0): > package_data['enum'].append( 'test/py3_test_enum.py') > else: > package_data['enum'].append( 'test/py2_test_enum.py') > > setup(package_data = package_data, ...) Cool! Is there a way to have files renamed? Ideally the installed package would just have enum.py, test_enum.py, and not py2_enum.py and py2_test_enum.py -- ~Ethan~ From ncoghlan at gmail.com Sat Jun 29 11:07:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jun 2013 19:07:17 +1000 Subject: [Distutils] complicated setup In-Reply-To: <51CE259F.30000@stoneleaf.us> References: <51BD65A5.3040500@stoneleaf.us> <51CC76F5.3030208@stoneleaf.us> <51CE201B.7070509@stoneleaf.us> <51CE259F.30000@stoneleaf.us> Message-ID: On 29 June 2013 10:09, Ethan Furman wrote: > Am I making this too complicated? Can I just do my renaming in the setup.py > script before calling the setup function? You can't easily create a wheel that way. What would actually be better is if you could avoid the need for any Python 3 specific syntax in the first place. Are you sure you can't tweak the code to use types.new_class or an inner function call to avoid the syntax problems? For example, Python 2 can do keyword only arguments like this: def _unpack_args(module=None, type=None): return module, type class CallableWithKeywordOnlyArgs def __call__(cls, value, names=None, **kwargs): module, type = _unpack_kwargs(**kwargs) The introspection support and error messages aren't as good as those for true Python 3 keyword-only arguments, but they're not *that* bad. There's a reason shared syntax compatible 2/3 source has become the most popular approach for straddling the 2/3 boundary - the alternatives are all lousy by comparison. Conditional distribution of version specific source files is far more painful than jumping through a few syntactic hoops to stay within the common 2/3 subset of the language. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Sat Jun 29 14:26:20 2013 From: holger at merlinux.eu (holger krekel) Date: Sat, 29 Jun 2013 12:26:20 +0000 Subject: [Distutils] Download Counts on PyPI In-Reply-To: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> References: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> Message-ID: <20130629122620.GD12023@merlinux.eu> On Thu, Jun 27, 2013 at 17:18 -0400, Donald Stufft wrote: > Download counts have (kind of) been re-enabled on PyPI. > > The new download counts work as so: > > * At the bottom of a detail page there is rolling tallies for the last > day, the last week, and the last month[1] > * The API exposes total download counts per file still. > * The rolling counts update more or less in real time while the total counts in the API update hourly. > * We no longer incorporate download counts from mirrors, making the mirroring infrastructure a tree and not a federation. > > [1] Only the last day is enabled currently, once more data is available the last week and last month will also be enabled. haven't investigated much but FYI it appears to me that the download numbers are not accurate. Yesterday i downloaded a package myself three times alone from three different locations and the download number i see now is "2". And i am pretty sure i wasn't the only one doing a download. cheers, holger > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From aclark at aclark.net Sat Jun 29 14:51:01 2013 From: aclark at aclark.net (Alex Clark) Date: Sat, 29 Jun 2013 12:51:01 +0000 (UTC) Subject: [Distutils] Download Counts on PyPI References: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> <20130629122620.GD12023@merlinux.eu> Message-ID: holger krekel merlinux.eu> writes: > Yesterday i downloaded a package myself three times alone from three different locations and the download number i see now is "2". And i am pretty sure i wasn't the only one doing a download. What package? From holger at merlinux.eu Sat Jun 29 15:00:49 2013 From: holger at merlinux.eu (holger krekel) Date: Sat, 29 Jun 2013 13:00:49 +0000 Subject: [Distutils] Download Counts on PyPI In-Reply-To: References: <00DF4887-2491-4AAD-9D4B-9E9EF399B777@stufft.io> <20130629122620.GD12023@merlinux.eu> Message-ID: <20130629130049.GH12023@merlinux.eu> On Sat, Jun 29, 2013 at 12:51 +0000, Alex Clark wrote: > holger krekel merlinux.eu> writes: > > Yesterday i downloaded a package myself three times alone from three > different locations and the download number i see now is "2". And i am > pretty sure i wasn't the only one doing a download. > > > What package? I was using vanity (good name, btw :) and for example looking at devpi-0.9.tar.gz shows "0" downloads. Also, devpi-server saw a few hundred downloads per release before the CDN move and now shows up with "1" or so. Given that i just gave a training where a few people had it installed i am pretty sure that can't be. cheers, holger From ethan at stoneleaf.us Sat Jun 29 21:08:23 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 29 Jun 2013 12:08:23 -0700 Subject: [Distutils] complicated setup In-Reply-To: References: <51BD65A5.3040500@stoneleaf.us> <51CC76F5.3030208@stoneleaf.us> <51CE201B.7070509@stoneleaf.us> <51CE259F.30000@stoneleaf.us> Message-ID: <51CF30A7.6000004@stoneleaf.us> On 06/29/2013 02:07 AM, Nick Coghlan wrote: > On 29 June 2013 10:09, Ethan Furman wrote: >> Am I making this too complicated? Can I just do my renaming in the setup.py >> script before calling the setup function? > > You can't easily create a wheel that way. > > What would actually be better is if you could avoid the need for any > Python 3 specific syntax in the first place. Are you sure you can't > tweak the code to use types.new_class or an inner function call to > avoid the syntax problems? > > For example, Python 2 can do keyword only arguments like this: > > def _unpack_args(module=None, type=None): > return module, type > > class CallableWithKeywordOnlyArgs > def __call__(cls, value, names=None, **kwargs): > module, type = _unpack_kwargs(**kwargs) > > The introspection support and error messages aren't as good as those > for true Python 3 keyword-only arguments, but they're not *that* bad. > > There's a reason shared syntax compatible 2/3 source has become the > most popular approach for straddling the 2/3 boundary - the > alternatives are all lousy by comparison. Conditional distribution of > version specific source files is far more painful than jumping through > a few syntactic hoops to stay within the common 2/3 subset of the > language. I was hoping to provide good examples of Python 3 code (as opposed to good examples of 2/3 boundary straddling), but yeah, it's danged difficult! Is there a way to have both a Py2 distribution and a Py3 distribution available on PyPI? -- ~Ethan~ From ncoghlan at gmail.com Sun Jun 30 01:14:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Jun 2013 09:14:12 +1000 Subject: [Distutils] Upcoming changes to PEP 426/440 Message-ID: Donald has been continuing his data modelling work for Warehouse (aka PyPI 2.0: https://github.com/dstufft/warehouse) and found that the *_requires/*_may_require split for dependencies was significantly more painful to work with than I had expected. Accordingly, I'm making some adjustments to the way dependencies are defined to bring them more into line with the way the contributors field and contact metadata works: 1. The "*_may_require" fields are all going away (leaving only the "*_requires" fields) 2. The "*_requires" fields are becoming lists of "dependency specifier" mappings rather than strings 3. A dependency specifier is now a mapping with the following fields: * "install": the installation specifier for the dependency * "extra": as per the current PEP (for conditional dependencies) * "environment": as per the current PEP (for conditional dependencies) 4. The "install" subfield is compulsory, the other two are optional (as now, using either of the latter creates a "conditional dependency", while dependency declarations with only the "install" subfield are unconditional) 5. An installation specifier is what PEP 426 currently calls a dependency specifier: the "name [extras] (constraints)" format. They will get their own top level section (similar to the existing Extras and Environment markers sections) In addition to those changes, I'll also be making a change to the recommended handling of virtual distributions (again based on Donald's feedback): integration tools should NEVER look at the "Provides" listings on a public index server to satisfy unmet dependencies. Instead, virtual distributions should be defined in such a way that whoever first invents the specific virtual distribution name registers it on PyPI, using the dependency metadata to pull in a default implementation. That's the only way to manage virtual distributions on a public index server that isn't vulnerable to later hijacking simply by registering a distribution with that name. As part of documenting that, I'll probably give the notion of "Virtual distributions" their own top level section (these are distributions that don't have any code of their own - they just declare dependencies on other projects to make them easy to install as a group, or to define the default provider for a dependency that may be satisfied by any one of multiple distributions). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Jun 30 01:18:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Jun 2013 09:18:01 +1000 Subject: [Distutils] complicated setup In-Reply-To: <51CF30A7.6000004@stoneleaf.us> References: <51BD65A5.3040500@stoneleaf.us> <51CC76F5.3030208@stoneleaf.us> <51CE201B.7070509@stoneleaf.us> <51CE259F.30000@stoneleaf.us> <51CF30A7.6000004@stoneleaf.us> Message-ID: On 30 June 2013 05:08, Ethan Furman wrote: > I was hoping to provide good examples of Python 3 code (as opposed to good > examples of 2/3 boundary straddling), but yeah, it's danged difficult! Writing idiomatic Python 3 code and supporting both Python 2 & 3 aren't really compatible goals. I suggest pointing to the stdlib implementation for the idiomatic Python 3 only code, and just targeting the common 2/3 subset for the PyPI version - a lot of people have been down this road before you and it really is the easiest way to do it :) > Is there a way to have both a Py2 distribution and a Py3 distribution > available on PyPI? Register two different names (which also makes for a fairly poor end user experience) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Sun Jun 30 09:44:35 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 30 Jun 2013 07:44:35 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: Message-ID: Nick Coghlan gmail.com> writes: > Donald has been continuing his data modelling work for Warehouse (aka > PyPI 2.0: https://github.com/dstufft/warehouse) and found that the > *_requires/*_may_require split for dependencies was significantly more > painful to work with than I had expected. Has there been any public discussion about this? I'm just curious about what the difficulties were. > 1. The "*_may_require" fields are all going away (leaving only the > "*_requires" fields) > > 2. The "*_requires" fields are becoming lists of "dependency > specifier" mappings rather than strings I'm wondering if this area could be simplified further. For example, can't we lose test_requires, meta_requires, build_requires and dev_requires just by stating that "test", "meta", "build" and "dev" are reserved extra names which don't need to be explicitly defined in "extras"? Then you get just one list of dependency specifiers, which can be readily filtered to provide what is currently provided by {test,meta,build,dev}_requires. It seems to lead to a very simple data model, as well as making the JSON schema more concise. Regards, Vinay Sajip From ncoghlan at gmail.com Sun Jun 30 10:00:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Jun 2013 18:00:57 +1000 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: On 30 June 2013 17:44, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> Donald has been continuing his data modelling work for Warehouse (aka >> PyPI 2.0: https://github.com/dstufft/warehouse) and found that the >> *_requires/*_may_require split for dependencies was significantly more >> painful to work with than I had expected. > > Has there been any public discussion about this? I'm just curious about what > the difficulties were. > >> 1. The "*_may_require" fields are all going away (leaving only the >> "*_requires" fields) >> >> 2. The "*_requires" fields are becoming lists of "dependency >> specifier" mappings rather than strings > > I'm wondering if this area could be simplified further. For example, can't > we lose test_requires, meta_requires, build_requires and dev_requires just > by stating that "test", "meta", "build" and "dev" are reserved extra names > which don't need to be explicitly defined in "extras"? Then you get just one > list of dependency specifiers, which can be readily filtered to provide what > is currently provided by {test,meta,build,dev}_requires. It seems to lead to > a very simple data model, as well as making the JSON schema more concise. No, because the semantic dependencies form a Cartesian product with the extras. You can define :meta:, :run:, :test:, :build: and :dev: dependencies for any extra. So if, for example, you have a "localweb" extra, then you can define extra test dependencies for that. The semantic specifiers determine *which sets of dependencies* you're interested in, while the explicit extras define optional subsets of those. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Sun Jun 30 10:53:27 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 30 Jun 2013 08:53:27 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: Message-ID: Nick Coghlan gmail.com> writes: > No, because the semantic dependencies form a Cartesian product with > the extras. You can define :meta:, :run:, :test:, :build: and :dev: > dependencies for any extra. So if, for example, you have a "localweb" > extra, then you can define extra test dependencies for that. > > The semantic specifiers determine *which sets of dependencies* you're > interested in, while the explicit extras define optional subsets of > those. Isn't that the same as having an additional field in the dependency mapping? It seems like that's how one would organise it at the RDBMS level, anyway. { "install": "localweb-test-util [win] (>= 1.0)", "extra": "localweb", "environment": "sys_platform == 'win32'", "kind": ":test:" } Sorry if I'm being dense :-( Regards, Vinay Sajip From ncoghlan at gmail.com Sun Jun 30 13:21:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Jun 2013 21:21:54 +1000 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: On 30 June 2013 18:53, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> No, because the semantic dependencies form a Cartesian product with >> the extras. You can define :meta:, :run:, :test:, :build: and :dev: >> dependencies for any extra. So if, for example, you have a "localweb" >> extra, then you can define extra test dependencies for that. >> >> The semantic specifiers determine *which sets of dependencies* you're >> interested in, while the explicit extras define optional subsets of >> those. > > Isn't that the same as having an additional field in the dependency mapping? > It seems like that's how one would organise it at the RDBMS level, anyway. > > { > "install": "localweb-test-util [win] (>= 1.0)", > "extra": "localweb", > "environment": "sys_platform == 'win32'", > "kind": ":test:" > } You certainly *could* define it like that, but no existing dependency system I'm aware of does it that way. If they allow for anything other than runtime dependencies in the first place, they define a different top level field: * setuptools has requires and install_requires * PEP 346 has Requires-Dist and Setup-Requires-Dist * RPM has Requires and BuildRequires * npm has dependencies and devDependencies The different kinds of semantic dependency make fundamentally different statements about the distributions being referenced. In particular, automated tools (especially PyPI) may place different constraints on the kind of specifier they allow in each field. PEP 426 explicitly *requires* that runtime dependencies be split between :meta: (which requires exact version specifiers) and :run: (which disallows them) so tools can issue appropriate warnings or errors when someone pins a specific version without explicitly stating their intent to monitor that dependency responsibly. That kind of difference in what's appropriate indicates they're only the "same thing" at a superficial syntactic level. I decided to merge the *_requires and *_may_require fields because they had syntactic differences that made it hard to do unified processing on them. That's what Donald noticed and pointed out to me off-list - the need for two distinct code paths just to deal with the fact that *_requires used strings while *_may_require used a mapping with a nested list, even when the natural model in the database for an unconditional dependency is the same as a conditional dependency with a NULL extra and environment field. Having them split as they currently are in the PEP is also directly responsible for the odd "must define extra, environment or both" constraint on the "*_may_require" field entries. Merging the fields eliminates all that complexity - the only required subfield is now "install", with the two conditional fields both optional. This means unconditional dependencies can be read and written using the same code path as conditional dependencies, since the conditional dependency code already needed to cope with the fact the "extra" or "environment" subfields might be missing. We don't get the same level of payoff by switching to a "kind" subfield, because all five dependency fields already use the same internal syntax. Whether you're keying off the top level field name, or keying off a "kind" subfield, the processing code will still be identical across all five kinds of dependency. As far as data modelling goes, Warehouse actually splits the different kinds of dependency as distinct ORM classes. This allows various useful details (like the descriptive names for each kind of dependency) to be handled directly in the data model, rather than needing to be tracked separately. While the two forms are functionally equivalent, I still prefer multiple top level fields, as I consider it both easier to document and more consistent with the approach used by other packaging systems. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Jun 30 18:47:16 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Jun 2013 12:47:16 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: On Jun 30, 2013, at 7:21 AM, Nick Coghlan wrote: > On 30 June 2013 18:53, Vinay Sajip wrote: >> Nick Coghlan gmail.com> writes: >> >>> No, because the semantic dependencies form a Cartesian product with >>> the extras. You can define :meta:, :run:, :test:, :build: and :dev: >>> dependencies for any extra. So if, for example, you have a "localweb" >>> extra, then you can define extra test dependencies for that. >>> >>> The semantic specifiers determine *which sets of dependencies* you're >>> interested in, while the explicit extras define optional subsets of >>> those. >> >> Isn't that the same as having an additional field in the dependency mapping? >> It seems like that's how one would organise it at the RDBMS level, anyway. >> >> { >> "install": "localweb-test-util [win] (>= 1.0)", >> "extra": "localweb", >> "environment": "sys_platform == 'win32'", >> "kind": ":test:" >> } > > > We don't get the same level of payoff by switching to a "kind" > subfield, because all five dependency fields already use the same > internal syntax. Whether you're keying off the top level field name, > or keying off a "kind" subfield, the processing code will still be > identical across all five kinds of dependency. > > As far as data modelling goes, Warehouse actually splits the different > kinds of dependency as distinct ORM classes. This allows various > useful details (like the descriptive names for each kind of > dependency) to be handled directly in the data model, rather than > needing to be tracked separately. > > While the two forms are functionally equivalent, I still prefer > multiple top level fields, as I consider it both easier to document > and more consistent with the approach used by other packaging systems. > Using a kind subfield actually is harder and requires more work. There's no way around needing to process each conditional dependency and check if they match, however each "kind" is always going to be an all or nothing kind of deal. You either want to process all of the dependencies of a certain kind, or none of them. As it stands right now you can just unconditionally loop over each dependency in a run_requires. However if all there was was required, you'd need to look over the entire list and check the kind subfield. Even worse if the order you install a "kind" matters (e.g. need to install build_requires prior to install run_requires) you'll need to loop over the list multiple times. I pushed for this change for the same basic reason I'm against the change you're mentioning. I think in order to make things easier for processing the first thing a tool would do given a single unified list with a subfield of "kind", is split them into several variables (wether variables in their own right, or as keys in a dictionary). If the natural inclination is to split them, we should just split them up front and make things simpler. Similarly I felt that it was more natural for a tool to want to condense the *_requires and *_may_requires so that they could easily run it though a single codepath without needing conditionals scattered all over. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Sun Jun 30 21:11:59 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 30 Jun 2013 15:11:59 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: Sun, Jun 30, 2013 at 12:47 PM, Donald Stufft wrote: > > On Jun 30, 2013, at 7:21 AM, Nick Coghlan wrote: > >> On 30 June 2013 18:53, Vinay Sajip wrote: >>> Nick Coghlan gmail.com> writes: >>> >>>> No, because the semantic dependencies form a Cartesian product with >>>> the extras. You can define :meta:, :run:, :test:, :build: and :dev: >>>> dependencies for any extra. So if, for example, you have a "localweb" >>>> extra, then you can define extra test dependencies for that. >>>> >>>> The semantic specifiers determine *which sets of dependencies* you're >>>> interested in, while the explicit extras define optional subsets of >>>> those. >>> >>> Isn't that the same as having an additional field in the dependency mapping? >>> It seems like that's how one would organise it at the RDBMS level, anyway. >>> >>> { >>> "install": "localweb-test-util [win] (>= 1.0)", >>> "extra": "localweb", >>> "environment": "sys_platform == 'win32'", >>> "kind": ":test:" >>> } >> >> >> We don't get the same level of payoff by switching to a "kind" >> subfield, because all five dependency fields already use the same >> internal syntax. Whether you're keying off the top level field name, >> or keying off a "kind" subfield, the processing code will still be >> identical across all five kinds of dependency. >> >> As far as data modelling goes, Warehouse actually splits the different >> kinds of dependency as distinct ORM classes. This allows various >> useful details (like the descriptive names for each kind of >> dependency) to be handled directly in the data model, rather than >> needing to be tracked separately. >> >> While the two forms are functionally equivalent, I still prefer >> multiple top level fields, as I consider it both easier to document >> and more consistent with the approach used by other packaging systems. >> > > Using a kind subfield actually is harder and requires more work. There's no way around needing to process each conditional dependency and check if they match, however each "kind" is always going to be an all or nothing kind of deal. You either want to process all of the dependencies of a certain kind, or none of them. As it stands right now you can just unconditionally loop over each dependency in a run_requires. However if all there was was required, you'd need to look over the entire list and check the kind subfield. Even worse if the order you install a "kind" matters (e.g. need to install build_requires prior to install run_requires) you'll need to loop over the list multiple times. It is the same amount of hard. The dependency resolution system would probably want to build an identical requirements[kind][extra] data structure once no matter what the input looked liked by looping over the very short list a single time, or perhaps expand the extra names to all start with ":run:" + extra_name etc. The most important thing would be to avoid having the actual installer code do anything different based on the category of dependency so that pip doesn't wind up with 5 different install commands. > I pushed for this change for the same basic reason I'm against the change you're mentioning. I think in order to make things easier for processing the first thing a tool would do given a single unified list with a subfield of "kind", is split them into several variables (wether variables in their own right, or as keys in a dictionary). If the natural inclination is to split them, we should just split them up front and make things simpler. Similarly I felt that it was more natural for a tool to want to condense the *_requires and *_may_requires so that they could easily run it though a single codepath without needing conditionals scattered all over. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From vinay_sajip at yahoo.co.uk Sun Jun 30 22:26:15 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 30 Jun 2013 20:26:15 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: Message-ID: Donald Stufft stufft.io> writes: > > While the two forms are functionally equivalent, I still prefer > > multiple top level fields, as I consider it both easier to document > > and more consistent with the approach used by other packaging systems. > > > Using a kind subfield actually is harder and requires more work. I take Nick's points regarding his preference and the reasons for it, but I'm not convinced by "harder". Having actually implemented the processing logic in distlib, it didn't seem especially hard to me. We already have more "kinds" than the other dependency systems Nick mentioned: one advantage of a single list of the kind I described is that you could in theory add additional "kinds", for example ":doc:", without much impact. Not that I'm arguing for any such addition - I'm quite aware of YAGNI. However, the fact that the single-list offers such possibilities is satisfying (as in, it feels right), and having implemented the multiple-list solution in distlib already, it just struck me that a single list would be more elegant. Leaving ORMs aside, I'm pretty sure if I was just working at the RDBMS layer, this is how I would structure the dependencies - using a single "dependency" table. It's a matter of detail as to exactly how a tool would process the single list - I'm reasonably confident that a readable/understandable and sufficiently performant implementation is achievable using a single pass over the JSON, and furthermore, worrying about multiple passes at this stage feels like premature optimisation. Regards, Vinay Sajip From donald at stufft.io Sun Jun 30 22:37:09 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Jun 2013 16:37:09 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: Message-ID: <88046B70-B23B-4DC1-98BF-75F01D58203C@stufft.io> On Jun 30, 2013, at 4:26 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >>> While the two forms are functionally equivalent, I still prefer >>> multiple top level fields, as I consider it both easier to document >>> and more consistent with the approach used by other packaging systems. >>> > >> Using a kind subfield actually is harder and requires more work. > > I take Nick's points regarding his preference and the reasons for it, but > I'm not convinced by "harder". Having actually implemented the processing > logic in distlib, it didn't seem especially hard to me. We already have more > "kinds" than the other dependency systems Nick mentioned: one advantage of a > single list of the kind I described is that you could in theory add > additional "kinds", for example ":doc:", without much impact. Not that I'm > arguing for any such addition - I'm quite aware of YAGNI. However, the fact > that the single-list offers such possibilities is satisfying (as in, it > feels right), and having implemented the multiple-list solution in distlib > already, it just struck me that a single list would be more elegant. Leaving > ORMs aside, I'm pretty sure if I was just working at the RDBMS layer, this > is how I would structure the dependencies - using a single "dependency" table. > > It's a matter of detail as to exactly how a tool would process the single > list - I'm reasonably confident that a readable/understandable and > sufficiently performant implementation is achievable using a single pass > over the JSON, and furthermore, worrying about multiple passes at this stage > feels like premature optimization. I'm not worried about the speed or performance, I'm worried about how annoying it is to write a tool that processes it. I optimize for the API it presents. Given your suggestion the first thing I'd do is take the single list and split them into multiple lists for easier processing. It's the (IMO) natural thing to do given a single list like that given the phased nature of installation. If it feels natural to split it, then you might as well split it up front and save the implementers from needing to do so. You might disagree, and that's fine. But it's not a performance based argument. As to the other question of adding new things, you can add a new top level keyword just as easily too in a completely backwards compatible fashion. If I was nitpicking i'd stick it in "dependencies": {"runtime": [?], "build": [?]} just for organizations sake. But I'm perfectly happy with them being top level key words too. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Sun Jun 30 22:47:16 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 30 Jun 2013 20:47:16 +0000 (UTC) Subject: [Distutils] Upcoming changes to PEP 426/440 References: <88046B70-B23B-4DC1-98BF-75F01D58203C@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > I'm not worried about the speed or performance, I'm worried about how > annoying it is to write a tool that processes it. Quite possibly I misunderstood the thrust of your argument, but the point of distlib is to take care of those kinds of low-level details, rather than having multiple packaging tools reimplement that processing you refer to. Of course people are free to not use distlib, but if I'm wasting my time then it would be nice if people would tell me :-) Regards, Vinay Sajip From donald at stufft.io Sun Jun 30 23:00:32 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Jun 2013 17:00:32 -0400 Subject: [Distutils] Upcoming changes to PEP 426/440 In-Reply-To: References: <88046B70-B23B-4DC1-98BF-75F01D58203C@stufft.io> Message-ID: <13C523C7-4A6C-4A5C-A4CA-CA2D1B5A8B58@stufft.io> On Jun 30, 2013, at 4:47 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I'm not worried about the speed or performance, I'm worried about how >> annoying it is to write a tool that processes it. > > Quite possibly I misunderstood the thrust of your argument, but the point of > distlib is to take care of those kinds of low-level details, rather than > having multiple packaging tools reimplement that processing you refer to. Of > course people are free to not use distlib, but if I'm wasting my time then > it would be nice if people would tell me :-) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig I'm not knocking the work you've done on distlib at all :) I just don't think it's existence means we shouldn't worry about the user friendliness of the API if someone doesn't use it. Maybe you think a single list is more user-friendly. That's fine for you to think that, we can agree to disagree on that :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: