From alexis at notmyidea.org Sun May 1 16:55:37 2011 From: alexis at notmyidea.org (=?UTF-8?B?QWxleGlzIE3DqXRhaXJlYXU=?=) Date: Sun, 01 May 2011 15:55:37 +0100 Subject: [Catalog-sig] PEP 345 - Environment markers and platform.python_implementation In-Reply-To: References: <4DBAEBFD.1040508@notmyidea.org> <4DBC0D2F.3060709@notmyidea.org> Message-ID: <4DBD7469.9010301@notmyidea.org> On 30/04/2011 14:38, Tarek Ziad? wrote: > I am +1 with the change, but I just need to see if we can do slight > changes to the PEP or we need a new PEP > > will keep you informed. It turned out it is possible to make slight changes to the PEP, so I added the support of this new environment marker in PEP 345 [1] and in packaging [2]. [1] http://hg.python.org/peps/rev/a09a267ab612 [2] https://bitbucket.org/tarek/cpython/changeset/81efc31169d2 -- Alexis From m.van.rees at zestsoftware.nl Mon May 2 18:12:28 2011 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Mon, 02 May 2011 18:12:28 +0200 Subject: [Catalog-sig] Some pypi mirrors not up to date? Message-ID: Hi, I noticed that some distributions are not on all mirrors. For example http://a.pypi.python.org/simple/plone.app.referenceablebehavior/ has 0.1 and 0.2 (last one released 30 April) but 0.2 is missing from http://b.pypi.python.org/simple/plone.app.referenceablebehavior/ Same for c and d. Ah, no: those two have it now. I know for sure that at least d did not have it five minutes ago. And this version has been released two days ago, so it should have been slightly faster. :-) Same is true for example for plone.dexterity: also missing on only b.pypi.python.org currently. Any idea what could be amiss? -- Maurits van Rees Web App Programmer at Zest Software: http://zestsoftware.nl Personal website: http://maurits.vanrees.org/ From ziade.tarek at gmail.com Mon May 2 18:16:51 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 2 May 2011 18:16:51 +0200 Subject: [Catalog-sig] Some pypi mirrors not up to date? In-Reply-To: References: Message-ID: On Mon, May 2, 2011 at 6:12 PM, Maurits van Rees wrote: > Hi, > > I noticed that some distributions are not on all mirrors. ?For example > http://a.pypi.python.org/simple/plone.app.referenceablebehavior/ > has 0.1 and 0.2 (last one released 30 April) > but 0.2 is missing from > http://b.pypi.python.org/simple/plone.app.referenceablebehavior/ > > Same for c and d. ?Ah, no: those two have it now. ?I know for sure that at > least d did not have it five minutes ago. ?And this version has been > released two days ago, so it should have been slightly faster. :-) > > Same is true for example for plone.dexterity: also missing on only > b.pypi.python.org currently. > > Any idea what could be amiss? This mirror is not up-to-date according to its last modified date: http://b.pypi.python.org/last-modified > > > -- > Maurits van Rees > Web App Programmer at Zest Software: http://zestsoftware.nl > Personal website: http://maurits.vanrees.org/ > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Tarek Ziad? | http://ziade.org From jannis at leidel.info Mon May 2 19:24:04 2011 From: jannis at leidel.info (Jannis Leidel) Date: Mon, 2 May 2011 19:24:04 +0200 Subject: [Catalog-sig] Some pypi mirrors not up to date? In-Reply-To: References: Message-ID: <9B68E47E-1639-445D-AF5F-3791BB5CCF7D@leidel.info> On 02.05.2011, at 18:12, Maurits van Rees wrote: > Hi, > > I noticed that some distributions are not on all mirrors. For example > http://a.pypi.python.org/simple/plone.app.referenceablebehavior/ > has 0.1 and 0.2 (last one released 30 April) > but 0.2 is missing from > http://b.pypi.python.org/simple/plone.app.referenceablebehavior/ > > Same for c and d. Ah, no: those two have it now. I know for sure that at least d did not have it five minutes ago. And this version has been released two days ago, so it should have been slightly faster. :-) Hm, d doesn't seem to have the file on disk even thought it's on the simple page, see http://d.pypi.python.org/packages/source/p/plone.app.referenceablebehavior/ Martin: Anything I can do to make sure this doesn't happen again? Jannis From ziade.tarek at gmail.com Mon May 2 19:32:27 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 2 May 2011 19:32:27 +0200 Subject: [Catalog-sig] Some pypi mirrors not up to date? In-Reply-To: References: Message-ID: On Mon, May 2, 2011 at 6:16 PM, Tarek Ziad? wrote: > On Mon, May 2, 2011 at 6:12 PM, Maurits van Rees > wrote: >> Hi, >> >> I noticed that some distributions are not on all mirrors. ?For example >> http://a.pypi.python.org/simple/plone.app.referenceablebehavior/ >> has 0.1 and 0.2 (last one released 30 April) >> but 0.2 is missing from >> http://b.pypi.python.org/simple/plone.app.referenceablebehavior/ >> >> Same for c and d. ?Ah, no: those two have it now. ?I know for sure that at >> least d did not have it five minutes ago. ?And this version has been >> released two days ago, so it should have been slightly faster. :-) >> >> Same is true for example for plone.dexterity: also missing on only >> b.pypi.python.org currently. >> >> Any idea what could be amiss? > > This mirror is not up-to-date according to its last modified date: > > http://b.pypi.python.org/last-modified on a side note, clients should warn when mirrors are outdated like this. Not sure about the UX but a > 1 month old mirror should be blacklisted from the client-side -- Tarek Ziad? | http://ziade.org From lists at zopyx.com Mon May 2 19:56:13 2011 From: lists at zopyx.com (Andreas Jung) Date: Mon, 02 May 2011 19:56:13 +0200 Subject: [Catalog-sig] Some pypi mirrors not up to date? In-Reply-To: References: Message-ID: <4DBEF03D.3030002@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziad? wrote: > On Mon, May 2, 2011 at 6:16 PM, Tarek Ziad? wrote: >> On Mon, May 2, 2011 at 6:12 PM, Maurits van Rees >> wrote: >>> Hi, >>> >>> I noticed that some distributions are not on all mirrors. For example >>> http://a.pypi.python.org/simple/plone.app.referenceablebehavior/ >>> has 0.1 and 0.2 (last one released 30 April) >>> but 0.2 is missing from >>> http://b.pypi.python.org/simple/plone.app.referenceablebehavior/ >>> >>> Same for c and d. Ah, no: those two have it now. I know for sure that at >>> least d did not have it five minutes ago. And this version has been >>> released two days ago, so it should have been slightly faster. :-) >>> >>> Same is true for example for plone.dexterity: also missing on only >>> b.pypi.python.org currently. >>> >>> Any idea what could be amiss? >> This mirror is not up-to-date according to its last modified date: >> >> http://b.pypi.python.org/last-modified > > on a side note, clients should warn when mirrors are outdated like this. > > Not sure about the UX but a > 1 month old mirror should be blacklisted > from the client-side I think the PyPI mirror monitoring procedure should take care about that. - -aj -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJNvvA9AAoJEADcfz7u4AZjJKMLwIBJ0b9UmY25LytsaDAs7zSN dcXWMQmA9VzmVmzk2EG2xCtq0HlFZlq0DsPKUwNMf8r6XmM+BYkvy3FuRTa94l/0 wXe8xy0F/8hQ9B04xUX0oohAZ4kOLsKnFkm2oXsWLjyyE0+5/+Y3PXT/qNVBneC+ M+GFpUt02AY3qQN+bwIHBZKFIhOBay2G0rJ9HHslB0sgeLuCb/yErB6oRggGftwE rBefXJddEN04zw6lKB+bfTCf2Yjki1RZEUGjmVJnJSGDJMbCXGQ3C5tCPILljduW 11etejosjrEr+Gke8DpTmWr0M3mIx1ckSfR+ihto1sPGifgPnISn82HLCATcjqeB WQOhAljj97PgOAgyMj0DArSzA8CHJj2ewOh//KZ5l6G1Ypel58zvLhfMSQdOomrA A0nUoo/sf5vjm1AYiHX359tJ1rM8XXPRQspqdtllD4XnOJ/ZmQe/UoHDzYiaioIF 3hi2M0PwX+N8HLuIJgeFFvsunwTIKLw= =wvCK -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From martin at v.loewis.de Mon May 2 22:10:28 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 02 May 2011 22:10:28 +0200 Subject: [Catalog-sig] Some pypi mirrors not up to date? In-Reply-To: <9B68E47E-1639-445D-AF5F-3791BB5CCF7D@leidel.info> References: <9B68E47E-1639-445D-AF5F-3791BB5CCF7D@leidel.info> Message-ID: <4DBF0FB4.5060900@v.loewis.de> Am 02.05.2011 19:24, schrieb Jannis Leidel: > > On 02.05.2011, at 18:12, Maurits van Rees wrote: > >> Hi, >> >> I noticed that some distributions are not on all mirrors. For example >> http://a.pypi.python.org/simple/plone.app.referenceablebehavior/ >> has 0.1 and 0.2 (last one released 30 April) >> but 0.2 is missing from >> http://b.pypi.python.org/simple/plone.app.referenceablebehavior/ >> >> Same for c and d. Ah, no: those two have it now. I know for sure that at least d did not have it five minutes ago. And this version has been released two days ago, so it should have been slightly faster. :-) > > Hm, d doesn't seem to have the file on disk even thought it's on the simple page, see http://d.pypi.python.org/packages/source/p/plone.app.referenceablebehavior/ > > Martin: Anything I can do to make sure this doesn't happen again? As the starting point, we should figure out why it happened in the first place - it shouldn't have, of course. Most likely, it's a bug :-) Regards, Martin From lists at zopyx.com Fri May 6 11:34:27 2011 From: lists at zopyx.com (Andreas Jung) Date: Fri, 06 May 2011 11:34:27 +0200 Subject: [Catalog-sig] c.pypi.python.org on the move. Message-ID: <4DC3C0A3.2080000@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I will move the c. mirror today to a new machine. Please update the DNS entry for the mirror during the next days to 46.163.78.160 Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJNw8CjAAoJEADcfz7u4AZjTVgLvArUYzcrJn+vt2zt5n/faxK/ zgAutZ+7k/cRhBwT9K5dU3gszFVB3hso3Hn9J3MwC7Um82UekVWwsagcqzCpt6Ha Wx2y5XAco1KGGfQXsXSkNZqB+nFRk/cFDw6gb40J03/4HObr0HOFbpfXZ0j5a9fN Y8lUMP2nkRO3hkMjvJnrSiRVgo3HcqhPy7XNz7xv4QpzdgPrqSyCEsW1s7V27nxR mwyavrrqk62KozIyK/9wlUe5CqIilNTsyGfyNTHmG4MrF95tXzAMmL8J9HXR6JhV Cn/65IggY2/QtvHGbFdojHadBU8WQStTIAvfkeXUGrJDGFmuic30YtEOlEOh9tDB BpRmOYbb1u6CVnEyoRfLTulJ7G5r2rSJPJHQu4CTVLdvGcqrDEARonzjgGpGXQPp C6PIwumU3dq+H94NJZ9gB68VOQ1+KODFfX+Ad9WtPH0lzhWlpeMpKuO60mw8cbY6 Ya/tQg4nRRzuSQBByAMgNCoghosEYhA= =IMk/ -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From stephane at harobed.org Mon May 9 11:10:45 2011 From: stephane at harobed.org (=?ISO-8859-1?Q?St=E9phane_Klein?=) Date: Mon, 09 May 2011 11:10:45 +0200 Subject: [Catalog-sig] I can't download elementtree from Pypi, what can I do ? Message-ID: Hi, I can't download elementtree from Pypi : http://pypi.python.org/pypi/elementtree/1.2.6-20050316 This resource is down http://effbot.org/downloads#elementtree What can I do ? I found this package here : ftp://64.50.236.52/.1/gentoo/distfiles/elementtree-1.2.6-20050316.tar.gz I've installed this package directly from this link. Some Pypi admin-sys can append this package directly in Pypi ? To avoid external link and fix correctly this issue ? Thanks for your comment. Regards, St?phane -- St?phane Klein blog: http://stephane-klein.info Twitter: http://twitter.com/klein_stephane pro: http://www.is-webdesign.com From ziade.tarek at gmail.com Mon May 9 11:27:41 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 9 May 2011 11:27:41 +0200 Subject: [Catalog-sig] I can't download elementtree from Pypi, what can I do ? In-Reply-To: References: Message-ID: On Mon, May 9, 2011 at 11:10 AM, St?phane Klein wrote: > Hi, > > I can't download elementtree from Pypi : > http://pypi.python.org/pypi/elementtree/1.2.6-20050316 > > This resource is down http://effbot.org/downloads#elementtree > > What can I do ? > I found this package here : > ftp://64.50.236.52/.1/gentoo/distfiles/elementtree-1.2.6-20050316.tar.gz > > I've installed this package directly from this link. > > Some Pypi admin-sys can append this package directly in Pypi ? > To avoid external link and fix correctly this issue ? > > Thanks for your comment. You should contact the author so he can fix the page. Also, you could suggest him to upload the releases at PyPI, this is highly recommended. But on our side, there's not much we can do if the url where the package should be found is down. We're can't upload people packages at PyPI for them. Cheers Tarek > > Regards, > St?phane > -- > St?phane Klein > blog: http://stephane-klein.info > Twitter: http://twitter.com/klein_stephane > pro: http://www.is-webdesign.com > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Mon May 9 12:21:45 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 09 May 2011 12:21:45 +0200 Subject: [Catalog-sig] I can't download elementtree from Pypi, what can I do ? In-Reply-To: References: Message-ID: <4DC7C039.9030404@v.loewis.de> > Some Pypi admin-sys can append this package directly in Pypi ? No, we won't. The author didn't give us permission to host the file, so we can't. > To avoid external link and fix correctly this issue ? The proper fix is to contact the author, and ask him to turn his server on again. Regards, Martin From ziade.tarek at gmail.com Thu May 12 17:57:20 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 12 May 2011 17:57:20 +0200 Subject: [Catalog-sig] PyPI's external packages Message-ID: Hey, I think some people are unaware of the fact that hosting themselves their packages can lead to problems when their websites are down. I'd like to propose these two very simple changes: - in packaging/distutils2, when the register command is called, just state that uploading the package would be a good idea :) - in pypi.python.org, on a project page that has no file uploaded, if the user connected is the project owner/maintainer, add a small message explaining why it's a good idea Maybe that could help reducing the number of external packages I'll definitely do something in distutils2 but maybe someone has a better idea ? Cheers Tarek -- Tarek Ziad? | http://ziade.org From jacob at jacobian.org Thu May 12 22:13:56 2011 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Thu, 12 May 2011 15:13:56 -0500 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: Message-ID: On Thu, May 12, 2011 at 10:57 AM, Tarek Ziad? wrote: > I think some people are unaware of the fact that hosting themselves > their packages can lead to problems when their websites are down. > > I'd like to propose these two very simple changes: Both sound fairly sensible to me. But why not do something a little less sensible, but a lot more useful? If the package is hosted externally grab a copy and host it locally. Then pip and friends can fall back to downloading from pypi if the original download URL is broken. Jacob From mal at egenix.com Thu May 12 23:44:15 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 12 May 2011 23:44:15 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: Message-ID: <4DCC54AF.1020106@egenix.com> Jacob Kaplan-Moss wrote: > On Thu, May 12, 2011 at 10:57 AM, Tarek Ziad? wrote: >> I think some people are unaware of the fact that hosting themselves >> their packages can lead to problems when their websites are down. >> >> I'd like to propose these two very simple changes: > > Both sound fairly sensible to me. > > But why not do something a little less sensible, but a lot more > useful? If the package is hosted externally grab a copy and host it > locally. Then pip and friends can fall back to downloading from pypi > if the original download URL is broken. If that's done in a way so that package authors can enable this functionality, that would be a possible solution as well. In the end, I think it's better to contact the package author to get the problem fixed. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-06-20: EuroPython 2011, Florence, Italy 39 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ben+python at benfinney.id.au Fri May 13 03:09:34 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 13 May 2011 11:09:34 +1000 Subject: [Catalog-sig] PyPI's external packages References: Message-ID: <87mxiriey9.fsf@benfinney.id.au> Jacob Kaplan-Moss writes: > But why not do something a little less sensible, but a lot more > useful? If the package is hosted externally grab a copy and host it > locally. Then pip and friends can fall back to downloading from pypi > if the original download URL is broken. That would require that PyPI accept packages only if the license terms allow PyPI to redistribute the work. I would support such a policy, but I think the PyPI administrator does not. I'd love to know otherwise. -- \ ?The Vatican is not a state.? a state must have people. There | `\ are no Vaticanians.? No-one gets born in the Vatican except by | _o__) an unfortunate accident.? ?Geoffrey Robertson, 2010-09-18 | Ben Finney From exarkun at twistedmatrix.com Thu May 12 21:04:48 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 12 May 2011 19:04:48 -0000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: Message-ID: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> On 03:57 pm, ziade.tarek at gmail.com wrote: >Hey, > >I think some people are unaware of the fact that hosting themselves >their packages can lead to problems when their websites are down. > >I'd like to propose these two very simple changes: > >- in packaging/distutils2, when the register command is called, just >state that uploading the package would be a good idea :) >- in pypi.python.org, on a project page that has no file uploaded, if >the user connected is the project owner/maintainer, add a small >message explaining why it's a good idea > >Maybe that could help reducing the number of external packages > >I'll definitely do something in distutils2 but maybe someone has a >better idea ? Make it easier to upload packages to PyPI. For example, add an scp- based interface or make "upload" work even if the package files exist on the filesystem somewhere already. Jean-Paul From exarkun at twistedmatrix.com Thu May 12 22:56:38 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 12 May 2011 20:56:38 -0000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> Message-ID: <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> On 07:21 pm, ziade.tarek at gmail.com wrote: >2011/5/12 : >>On 03:57 pm, ziade.tarek at gmail.com wrote: >>> >>>Hey, >>> >>>I think some people are unaware of the fact that hosting themselves >>>their packages can lead to problems when their websites are down. >>> >>>I'd like to propose these two very simple changes: >>> >>>- in packaging/distutils2, when the register command is called, just >>>state that uploading the package would be a good idea ?:) >>>- in pypi.python.org, on a project page that has no file uploaded, if >>>the user connected is the project owner/maintainer, add a small >>>message explaining why it's a good idea >>> >>>Maybe that could help reducing the number of external packages >>> >>>I'll definitely do something in distutils2 but maybe someone has a >>>better >>>idea ? >> >>Make it easier to upload packages to PyPI. ?For example, add an scp- >>based >>interface > >I think Martin added some ssh capability lately. Would make sense to >add it in distutils2. It's weird ssh stuff that so far hasn't seemed to make anything easier. I'm not entirely sure what its goal is. >> or make "upload" work even if the package files exist on the >>filesystem somewhere already. > >I am not sure to get that one. Like > >$ python setup.py upload /any/random/file ? Yes, like that. There are already server-side checks (which are too strict in at least one place, preventing legitimate files from being uploaded), so I don't see how it's a problem. Plus, if I really want to dump garbage onto PyPI, then I can still use the web interface. Making uploading inconvenient isn't a strategy for keeping trouble away. Jean-Paul From richard at python.org Fri May 13 08:35:27 2011 From: richard at python.org (Richard Jones) Date: Fri, 13 May 2011 16:35:27 +1000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> Message-ID: On 13 May 2011 06:56, wrote: > On 07:21 pm, ziade.tarek at gmail.com wrote: >> 2011/5/12 ?: >>> >>> On 03:57 pm, ziade.tarek at gmail.com wrote: >>>> >>>> Hey, >>>> >>>> I think some people are unaware of the fact that hosting themselves >>>> their packages can lead to problems when their websites are down. >>>> >>>> I'd like to propose these two very simple changes: >>>> >>>> - in packaging/distutils2, when the register command is called, just >>>> state that uploading the package would be a good idea ?:) >>>> - in pypi.python.org, on a project page that has no file uploaded, if >>>> the user connected is the project owner/maintainer, add a small >>>> message explaining why it's a good idea >>>> >>>> Maybe that could help reducing the number of external packages >>>> >>>> I'll definitely do something in distutils2 but maybe someone has a >>>> better >>>> idea ? >>> >>> Make it easier to upload packages to PyPI. ?For example, add an scp- >>> based >>> interface >> >> I think Martin added some ssh capability lately. Would make sense to >> add it in distutils2. > > It's weird ssh stuff that so far hasn't seemed to make anything easier. http://pypi.python.org/pypi/pypissh was developed to allow the distutils "upload" command to transmit the upload over ssh. Its intention isn't to make anything easier. It involves submitting an SSH key to PyPI but other than that it should just work - certainly not make anything harder. You're right about it being weird though - well, the heavy monkey-patching it does of distutils is anyway :-) > I'm not entirely sure what its goal is. How would your scp interface work? Do you have an existing implementation that you could refer to as a model? >>> ?or make "upload" work even if the package files exist on the >>> filesystem somewhere already. >> >> I am not sure to get that one. ?Like >> >> $ python setup.py upload /any/random/file ?? > > Yes, like that. ?There are already server-side checks (which are too strict > in at least one place, preventing legitimate files from being uploaded), so > I don't see how it's a problem. I'm not currently aware of any legitimate files being blocked at. There have been some issues in the past but I believe I'd be correct in saying that I can count the number of issues I've had to deal with on one hand. I do not believe we should allow uploading of arbitrary content as packages to PyPI. I'm not entirely comfortable with hosting the arbitrary content in the docs side of things, but that's because I'm way too paranoid about such things. Preventing re-uploading of files with the same name was done intentionally very early on to avoid end-user confusion and spurious bug reports (that is, people with distribution files of the same name but with different contents). > Plus, if I really want to dump garbage onto > PyPI, then I can still use the web interface. ?Making uploading inconvenient > isn't a strategy for keeping trouble away. The web form for uploading packages is subject to the same file legitimacy tests as the distutils upload command. They both use the same HTTP call on PyPI. Richard From exarkun at twistedmatrix.com Fri May 13 13:49:56 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 13 May 2011 11:49:56 -0000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> Message-ID: <20110513114956.1992.2123194803.divmod.xquotient.1344@localhost.localdomain> On 06:35 am, richard at python.org wrote: >On 13 May 2011 06:56, wrote: >>On 07:21 pm, ziade.tarek at gmail.com wrote: >>>2011/5/12 ?: >>>> >>>>On 03:57 pm, ziade.tarek at gmail.com wrote: >>>>> >>>>>Hey, >>>>> >>>>>I think some people are unaware of the fact that hosting themselves >>>>>their packages can lead to problems when their websites are down. >>>>> >>>>>I'd like to propose these two very simple changes: >>>>> >>>>>- in packaging/distutils2, when the register command is called, >>>>>just >>>>>state that uploading the package would be a good idea ?:) >>>>>- in pypi.python.org, on a project page that has no file uploaded, >>>>>if >>>>>the user connected is the project owner/maintainer, add a small >>>>>message explaining why it's a good idea >>>>> >>>>>Maybe that could help reducing the number of external packages >>>>> >>>>>I'll definitely do something in distutils2 but maybe someone has a >>>>>better >>>>>idea ? >>>> >>>>Make it easier to upload packages to PyPI. ?For example, add an scp- >>>>based >>>>interface >>> >>>I think Martin added some ssh capability lately. Would make sense to >>>add it in distutils2. >> >>It's weird ssh stuff that so far hasn't seemed to make anything >>easier. > >http://pypi.python.org/pypi/pypissh was developed to allow the >distutils "upload" command to transmit the upload over ssh. Its >intention isn't to make anything easier. It involves submitting an SSH >key to PyPI but other than that it should just work - certainly not >make anything harder. > >You're right about it being weird though - well, the heavy >monkey-patching it does of distutils is anyway :-) >>I'm not entirely sure what its goal is. > >How would your scp interface work? Do you have an existing >implementation that you could refer to as a model? >>>>?or make "upload" work even if the package files exist on the >>>>filesystem somewhere already. >>> >>>I am not sure to get that one. ?Like >>> >>>$ python setup.py upload /any/random/file ?? >> >>Yes, like that. ?There are already server-side checks (which are too >>strict >>in at least one place, preventing legitimate files from being >>uploaded), so >>I don't see how it's a problem. > >I'm not currently aware of any legitimate files being blocked at. There was one that I couldn't upload. I never figured out why, I just gave up on trying to distribute that file. Learning about file format type byte headers is also too high a barrier. >There have been some issues in the past but I believe I'd be correct >in saying that I can count the number of issues I've had to deal with >on one hand. > >I do not believe we should allow uploading of arbitrary content as >packages to PyPI. I'm not suggesting this. >[snip] >>Plus, if I really want to dump garbage onto >>PyPI, then I can still use the web interface. ?Making uploading >>inconvenient >>isn't a strategy for keeping trouble away. > >The web form for uploading packages is subject to the same file >legitimacy tests as the distutils upload command. They both use the >same HTTP call on PyPI. I don't think you understood what I was saying. The fact that the server imposes these checks is exactly why letting a user specify any file to "setup.py upload" is fine. The server can always reject it if it wants to. Jean-Paul From exarkun at twistedmatrix.com Fri May 13 13:59:15 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 13 May 2011 11:59:15 -0000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <20110513114956.1992.2123194803.divmod.xquotient.1344@localhost.localdomain> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> <20110513114956.1992.2123194803.divmod.xquotient.1344@localhost.localdomain> Message-ID: <20110513115915.1992.320942931.divmod.xquotient.1348@localhost.localdomain> On 11:49 am, exarkun at twistedmatrix.com wrote: >On 06:35 am, richard at python.org wrote: >>How would your scp interface work? Do you have an existing I don't know. Something like this, I guess: scp $PACKAGE_FILE pypi.python.org:~/$PROJECT/$RELEASE/ Jean-Paul From merwok at netwok.org Fri May 13 17:21:36 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 13 May 2011 17:21:36 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> Message-ID: <4DCD4C80.10803@netwok.org> Le 12/05/2011 21:04, exarkun at twistedmatrix.com a ?crit : > On 03:57 pm, ziade.tarek at gmail.com wrote: >> [...] >> I'll definitely do something in distutils2 but maybe someone has a >> better idea ? > Make it easier to upload packages to PyPI. For example, [...] > make "upload" work even if the package files exist on > the filesystem somewhere already. You?re referring to the fact that ?upload? won?t push files already present in dist/, right? I don?t know whether this was a design choice of Martin or merely done for code simplicity, but currently the command will only push files that are products of a command run from the same command line, e.g. ?sdist upload?. If you run ?sdist and then ?sdist upload?, is the sdist recreated even though there have been no changes to the files? If no, having to run ?sdist upload? is not inefficient, and makes you be explicit about the files you want to push, which is IMO good. Do you agree? If yes, I think it?s a bug. Regards From merwok at netwok.org Fri May 13 19:19:41 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 13 May 2011 19:19:41 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> Message-ID: <4DCD682D.8060801@netwok.org> Le 13/05/2011 19:12, exarkun at twistedmatrix.com a ?crit : > On 03:21 pm, merwok at netwok.org wrote: >> If you run ?sdist? and then ?sdist upload?, is the sdist >> recreated even though there have been no changes to the files? > Yes, the sdist is recreated. Arg, I thought the *dist commands were smarter. > I'm not concerned about the efficiency, though. I'm concerned about the > quality of the build. I want to generate a release, *test it*, and then > upload it. Being forced to generate it and upload it at the same time > gets in the way of this. Right. Now you have to run ?sdist?, test the result, and run ?sdist upload? hoping it will create the exact same file (I don?t see why it wouldn?t, but then again we?re talking about distutils , and there are also filesystem issues and all that). If this bug was fixed (the needless regeneration), would it be enough, or would you still like a way to upload an arbitrary file and let PyPI accept or reject it? (BTW, what kind of testing are you talking about? Unpackaging, running the test suite, installation? What tool are you using for that?) Regards From martin at v.loewis.de Sat May 14 02:15:28 2011 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 14 May 2011 02:15:28 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <4DCD4C80.10803@netwok.org> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> Message-ID: <4DCDC9A0.1060109@v.loewis.de> Am 13.05.2011 17:21, schrieb ?ric Araujo: > Le 12/05/2011 21:04, exarkun at twistedmatrix.com a ?crit : >> On 03:57 pm, ziade.tarek at gmail.com wrote: >>> [...] >>> I'll definitely do something in distutils2 but maybe someone has a >>> better idea ? >> Make it easier to upload packages to PyPI. For example, [...] >> make "upload" work even if the package files exist on >> the filesystem somewhere already. > > You?re referring to the fact that ?upload? won?t push files already > present in dist/, right? I don?t know whether this was a design choice > of Martin or merely done for code simplicity It's an explicit choice. Namely, the choice to have the uploader specify what kind of file this is, and what version of Python it is for, and what release it belongs to. The dist* commands have that information available. Short of changing the upload command, it should be possible to write a "existing_dist" command which accepts a file name, guesses file type and Python version, and then allows users to do python setup.py existing_dist --file foo.tgz upload Regards, Martin From tjreedy at udel.edu Sat May 14 05:09:45 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 13 May 2011 23:09:45 -0400 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: Message-ID: On 5/12/2011 11:57 AM, Tarek Ziad? wrote: > Hey, > > I think some people are unaware of the fact that hosting themselves > their packages can lead to problems when their websites are down. > > I'd like to propose these two very simple changes: > > - in packaging/distutils2, when the register command is called, just > state that uploading the package would be a good idea :) That is an opinion, not a fact. For some circumstances, it may not be a good idea. This whole discussion seems to ignore the license issue. Last I knew, PyPI would not accept uploads without agreeing to a license that some people find obnoxious and unacceptible. > - in pypi.python.org, on a project page that has no file uploaded, if > the user connected is the project owner/maintainer, add a small > message explaining why it's a good idea > > Maybe that could help reducing the number of external packages. Perhaps someone should ask people why they do not upload. > I'll definitely do something in distutils2 but maybe someone has a better idea ? -- Terry Jan Reedy From richard at python.org Sat May 14 07:48:47 2011 From: richard at python.org (Richard Jones) Date: Sat, 14 May 2011 15:48:47 +1000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <20110513114956.1992.2123194803.divmod.xquotient.1344@localhost.localdomain> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> <20110513114956.1992.2123194803.divmod.xquotient.1344@localhost.localdomain> Message-ID: On 13 May 2011 21:49, wrote: > On 06:35 am, richard at python.org wrote: > There was one that I couldn't upload. ?I never figured out why, I just gave > up on trying to distribute that file. ?Learning about file format type byte > headers is also too high a barrier. You shouldn't need to if it was produced by distutils. A file rejected in this way is a bug in PyPI. >> I do not believe we should allow uploading of arbitrary content as >> packages to PyPI. > > I'm not suggesting this. OK, I misunderstood. >> >> [snip] >>> >>> Plus, if I really want to dump garbage onto >>> PyPI, then I can still use the web interface. ?Making uploading >>> inconvenient >>> isn't a strategy for keeping trouble away. >> >> The web form for uploading packages is subject to the same file >> legitimacy tests as the distutils upload command. They both use the >> same HTTP call on PyPI. > > I don't think you understood what I was saying. ?The fact that the server > imposes these checks is exactly why letting a user specify any file to > "setup.py upload" is fine. ?The server can always reject it if it wants to. Yep, I understand your point now. Richard From stefan-usenet at bytereef.org Sat May 14 10:58:52 2011 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Sat, 14 May 2011 10:58:52 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: Message-ID: <20110514085852.GA16446@sleipnir.bytereef.org> Terry Reedy wrote: >> - in packaging/distutils2, when the register command is called, just >> state that uploading the package would be a good idea :) > > That is an opinion, not a fact. For some circumstances, it may not be a > good idea. I've uploaded my first package about half a year ago, so perhaps the following points might give some insight into the issues that new users encounter as well as the decisions they have to make. 1) The terms and conditions: As Terry mentioned, the terms and conditions are not very clear. I understood the intent only after reading VanL's comments in an old thread. Many package authors come from a culture that rejects licenses that require interpretation by a lawyer. As a consequence, they upload as little as possible. 2) SEO, content farms: When I uploaded the latest version of cdecimal in January, I had a long description on PyPI. I used some of the same phrases on my own website. Since PyPI is a juicy target for content scrapers, the text was duplicated all over the web within hours. Result: The original site http://www.bytereef.org/mpdecimal/index.html practically dropped from the Google index, while Softpedia etc. were doing fine. I waited a couple of weeks, then I removed the long description from PyPI. Within a couple of days the ranking of the original site was restored. [Note that the PyPI terms and conditions prevent me from going after content scrapers.] The point here is that content on PyPI can hurt the primary site. I don't know whether uploaded files contribute to this, but a package author might want to play it safe. 3) Control over download statistics: This one is obvious. 4) Primary site from the package user's point of view: If everything is on PyPI, users will start to regard PyPI as the primary site for a package (search engines already do). This might not be what a package author wants. 5) Other servers have good uptimes, too. Stefan Krah From ben+python at benfinney.id.au Sat May 14 16:23:29 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 15 May 2011 00:23:29 +1000 Subject: [Catalog-sig] PyPI's external packages References: <20110514085852.GA16446@sleipnir.bytereef.org> Message-ID: <87aaepicny.fsf@benfinney.id.au> Stefan Krah writes: > 1) The terms and conditions [?] > 2) SEO, content farms [?] > 3) Control over download statistics [?] > 4) Primary site from the package user's point of view [?] > 5) Other servers have good uptimes, too. Excellent set of points. Where before I couldn't see any good reason, I realise from this that there are many valid objections to hosting a package on PyPI. Thanks, Stefan, for making me think. -- \ ?If you always want the latest and greatest, then you have to | `\ buy a new iPod at least once a year.? ?Steve Jobs, MSNBC | _o__) interview 2006-05-25 | Ben Finney From richard at python.org Sun May 15 00:46:00 2011 From: richard at python.org (Richard Jones) Date: Sun, 15 May 2011 08:46:00 +1000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <20110514085852.GA16446@sleipnir.bytereef.org> References: <20110514085852.GA16446@sleipnir.bytereef.org> Message-ID: On 14 May 2011 18:58, Stefan Krah wrote: > Terry Reedy wrote: >>> - in packaging/distutils2, when the register command is called, just >>> state that uploading the package would be a good idea ?:) >> >> That is an opinion, not a fact. For some circumstances, it may not be a >> good idea. > > I've uploaded my first package about half a year ago, so perhaps the > following points might give some insight into the issues that new users > encounter as well as the decisions they have to make. > > 1) The terms and conditions: As Terry mentioned, the terms and conditions > ? are not very clear. I understood the intent only after reading VanL's > ? comments in an old thread. Many package authors come from a culture > ? that rejects licenses that require interpretation by a lawyer. As a > ? consequence, they upload as little as possible. This is interesting to me. Do you have a reference to that old posting of VanL's because I think I might have missed it. As far as I can tell there are no explicit terms and conditions on submissions to PyPI. It is stated in a couple of places that the code must be "open source" but I don't recall any serious discussion of the requirement. Richard From exarkun at twistedmatrix.com Sun May 15 03:58:15 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 15 May 2011 01:58:15 -0000 Subject: [Catalog-sig] PyPI upload-prevention bugs (was Re: PyPI's external packages) In-Reply-To: References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> <20110513114956.1992.2123194803.divmod.xquotient.1344@localhost.localdomain> Message-ID: <20110515015815.3983.319096476.divmod.xquotient.5@localhost.localdomain> On 14 May, 05:48 am, richard at python.org wrote: >On 13 May 2011 21:49, wrote: >>On 06:35 am, richard at python.org wrote: >>There was one that I couldn't upload. ?I never figured out why, I just >>gave >>up on trying to distribute that file. ?Learning about file format type >>byte >>headers is also too high a barrier. > >You shouldn't need to if it was produced by distutils. A file rejected >in this way is a bug in PyPI. Okay. There is such a bug. Is there a PyPI issue tracker I should report it in? Jean-Paul From stefan-usenet at bytereef.org Sun May 15 09:31:06 2011 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Sun, 15 May 2011 09:31:06 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: <20110514085852.GA16446@sleipnir.bytereef.org> Message-ID: <20110515073106.GA24384@sleipnir.bytereef.org> Richard Jones wrote: > On 14 May 2011 18:58, Stefan Krah wrote: > > 1) The terms and conditions: As Terry mentioned, the terms and conditions > > ? are not very clear. I understood the intent only after reading VanL's > > ? comments in an old thread. Many package authors come from a culture > > ? that rejects licenses that require interpretation by a lawyer. As a > > ? consequence, they upload as little as possible. > > This is interesting to me. Do you have a reference to that old posting > of VanL's because I think I might have missed it. As far as I can tell > there are no explicit terms and conditions on submissions to PyPI. It > is stated in a couple of places that the code must be "open source" > but I don't recall any serious discussion of the requirement. If you create an account, you have to agree to this: 3. "The PSF is free to use or disseminate any content that I upload on an unrestricted basis for any purpose. In particular, the PSF and all other users of the web site are granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form." Discussion of the usage agreement is in the thread starting here: http://mail.python.org/pipermail/catalog-sig/2009-December/002456.html VanL's comments are also in that thread. The terms are too broad. Because all users of PyPI get the same rights as the PSF, essentially every work that is uploaded gets an implicit CreativeCommons/no-deriv/no-attr stamp. M.-A. Lemburg has argued this in greater detail (see the paragraph under the link to the Google terms) here: http://mail.python.org/pipermail/catalog-sig/2009-December/002483.html Stefan Krah From exarkun at twistedmatrix.com Fri May 13 19:12:05 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 13 May 2011 17:12:05 -0000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <4DCD4C80.10803@netwok.org> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> Message-ID: <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> On 03:21 pm, merwok at netwok.org wrote: >Le 12/05/2011 21:04, exarkun at twistedmatrix.com a ?crit : >>On 03:57 pm, ziade.tarek at gmail.com wrote: >>>[...] >>>I'll definitely do something in distutils2 but maybe someone has a >>>better idea ? >>Make it easier to upload packages to PyPI. For example, [...] >>make "upload" work even if the package files exist on >>the filesystem somewhere already. > >You 19re referring to the fact that 1Cupload 1D won 19t push files already >present in dist/, right? More or less, yes. >I don 19t know whether this was a design choice >of Martin or merely done for code simplicity, but currently the command >will only push files that are products of a command run from the same >command line, e.g. 1Csdist upload 1D. If you run 1Csdist and then 1Csdist >upload 1D, is the sdist recreated even though there have been no changes >to the files? Yes, the sdist is recreated. > >If no, having to run 1Csdist upload 1D is not inefficient, and makes you >be >explicit about the files you want to push, which is IMO good. Do you >agree? > >If yes, I think it 19s a bug. I'm not concerned about the efficiency, though. I'm concerned about the quality of the build. I want to generate a release, *test it*, and then upload it. Being forced to generate it and upload it at the same time gets in the way of this. Jean-Paul From martin at v.loewis.de Mon May 16 00:28:53 2011 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 16 May 2011 00:28:53 +0200 Subject: [Catalog-sig] PyPI upload-prevention bugs (was Re: PyPI's external packages) In-Reply-To: <20110515015815.3983.319096476.divmod.xquotient.5@localhost.localdomain> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <20110512205638.1992.213027306.divmod.xquotient.1339@localhost.localdomain> <20110513114956.1992.2123194803.divmod.xquotient.1344@localhost.localdomain> <20110515015815.3983.319096476.divmod.xquotient.5@localhost.localdomain> Message-ID: <4DD053A5.2020804@v.loewis.de> >> You shouldn't need to if it was produced by distutils. A file rejected >> in this way is a bug in PyPI. > > Okay. There is such a bug. Is there a PyPI issue tracker I should > report it in? Follow the "Bug reports" link on PyPI, which leads you to http://sourceforge.net/tracker/?group_id=66150&atid=513503 Regards, Martin From tseaver at palladion.com Mon May 16 02:11:16 2011 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 15 May 2011 20:11:16 -0400 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/13/2011 01:12 PM, exarkun at twistedmatrix.com wrote: > On 03:21 pm, merwok at netwok.org wrote: >> Le 12/05/2011 21:04, exarkun at twistedmatrix.com a ?crit : >>> On 03:57 pm, ziade.tarek at gmail.com wrote: >>>> [...] >>>> I'll definitely do something in distutils2 but maybe someone has a >>>> better idea ? >>> Make it easier to upload packages to PyPI. For example, [...] >>> make "upload" work even if the package files exist on >>> the filesystem somewhere already. >> >> You 19re referring to the fact that 1Cupload 1D won 19t push files already >> present in dist/, right? > > More or less, yes. >> I don 19t know whether this was a design choice >> of Martin or merely done for code simplicity, but currently the command >> will only push files that are products of a command run from the same >> command line, e.g. 1Csdist upload 1D. If you run 1Csdist and then 1Csdist >> upload 1D, is the sdist recreated even though there have been no changes >> to the files? > > Yes, the sdist is recreated. >> >> If no, having to run 1Csdist upload 1D is not inefficient, and makes you >> be >> explicit about the files you want to push, which is IMO good. Do you >> agree? >> >> If yes, I think it 19s a bug. > > I'm not concerned about the efficiency, though. I'm concerned about the > quality of the build. I want to generate a release, *test it*, and then > upload it. Being forced to generate it and upload it at the same time > gets in the way of this. ISTM that if you run 'python setup.py sdist' a second time and get a different result (without changing any included files), then you have a built-in quality problem *in your setup.py.* Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3Qa6QACgkQ+gerLs4ltQ7tawCgjGf3C3Uvu8VB7Yp9ffr59gUW bJcAniOrvsiPRA8yhsyqNon1i653wHkv =1Jon -----END PGP SIGNATURE----- From exarkun at twistedmatrix.com Mon May 16 03:49:12 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 16 May 2011 01:49:12 -0000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> Message-ID: <20110516014912.3983.462173779.divmod.xquotient.26@localhost.localdomain> On 12:11 am, tseaver at palladion.com wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >On 05/13/2011 01:12 PM, exarkun at twistedmatrix.com wrote: >> >>I'm not concerned about the efficiency, though. I'm concerned about >>the >>quality of the build. I want to generate a release, *test it*, and >>then >>upload it. Being forced to generate it and upload it at the same time >>gets in the way of this. > >ISTM that if you run 'python setup.py sdist' a second time and get a >different result (without changing any included files), then you have a >built-in quality problem *in your setup.py.* Absolutely. I'd still rather avoid the possibility by not going that route. It's one less thing to worry about. And when it comes to setup.py, I already have plenty to worry about. Jean-Paul From exarkun at twistedmatrix.com Sun May 15 22:24:26 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 15 May 2011 20:24:26 -0000 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <4DCD682D.8060801@netwok.org> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> <4DCD682D.8060801@netwok.org> Message-ID: <20110515202426.3983.384657495.divmod.xquotient.12@localhost.localdomain> On 13 May, 05:19 pm, merwok at netwok.org wrote: >Le 13/05/2011 19:12, exarkun at twistedmatrix.com a ?crit : >>On 03:21 pm, merwok at netwok.org wrote: >>>If you run 1Csdist 1D and then 1Csdist upload 1D, is the sdist >>>recreated even though there have been no changes to the files? >>Yes, the sdist is recreated. > >Arg, I thought the *dist commands were smarter. >>I'm not concerned about the efficiency, though. I'm concerned about >>the >>quality of the build. I want to generate a release, *test it*, and >>then >>upload it. Being forced to generate it and upload it at the same time >>gets in the way of this. > >Right. Now you have to run 1Csdist 1D, test the result, and run 1Csdist >upload 1D hoping it will create the exact same file (I don 19t see why it >wouldn 19t, but then again we 19re talking about distutils , and >there >are also filesystem issues and all that). If this bug was fixed (the >needless regeneration), would it be enough, or would you still like a >way to upload an arbitrary file and let PyPI accept or reject it? > >(BTW, what kind of testing are you talking about? Unpackaging, running >the test suite, installation? What tool are you using for that?) Those kinds of things, yea. I don't have an automated tool for this. Jean-Paul From mal at egenix.com Mon May 16 11:44:55 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 16 May 2011 11:44:55 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> Message-ID: <4DD0F217.1050407@egenix.com> Tres Seaver wrote: > On 05/13/2011 01:12 PM, exarkun at twistedmatrix.com wrote: >> On 03:21 pm, merwok at netwok.org wrote: >>> Le 12/05/2011 21:04, exarkun at twistedmatrix.com a ?crit : >>>> On 03:57 pm, ziade.tarek at gmail.com wrote: >>>>> [...] >>>>> I'll definitely do something in distutils2 but maybe someone has a >>>>> better idea ? >>>> Make it easier to upload packages to PyPI. For example, [...] >>>> make "upload" work even if the package files exist on >>>> the filesystem somewhere already. >>> >>> You 19re referring to the fact that 1Cupload 1D won 19t push files already >>> present in dist/, right? > >> More or less, yes. >>> I don 19t know whether this was a design choice >>> of Martin or merely done for code simplicity, but currently the command >>> will only push files that are products of a command run from the same >>> command line, e.g. 1Csdist upload 1D. If you run 1Csdist and then 1Csdist >>> upload 1D, is the sdist recreated even though there have been no changes >>> to the files? > >> Yes, the sdist is recreated. >>> >>> If no, having to run 1Csdist upload 1D is not inefficient, and makes you >>> be >>> explicit about the files you want to push, which is IMO good. Do you >>> agree? >>> >>> If yes, I think it 19s a bug. > >> I'm not concerned about the efficiency, though. I'm concerned about the >> quality of the build. I want to generate a release, *test it*, and then >> upload it. Being forced to generate it and upload it at the same time >> gets in the way of this. > > ISTM that if you run 'python setup.py sdist' a second time and get a > different result (without changing any included files), then you have a > built-in quality problem *in your setup.py.* Just to clarify: creating an sdist really only means copying over the files from the MANIFEST into a temporary dir and then running tar or zip on the temporary directory. You can tell distutils to keep the temporary dir around by using the --keep-temp option on sdist. This will reduce the operation to just running the archiver on the temp dir again. The same can be done for bdist_* commands by having distutils skip the build process. Since I don't see how such trivial operations can affect the quality of the code, the discussion sounds a bit artificial :-) Note that if you want to do full testing of a release, you'll have to download the package from PyPI and then run the tests on the downloaded archive to be sure that things work - everything else is just preparation for this final run :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jjl at pobox.com Mon May 16 21:04:20 2011 From: jjl at pobox.com (John J Lee) Date: Mon, 16 May 2011 20:04:20 +0100 (BST) Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <4DD0F217.1050407@egenix.com> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> <4DD0F217.1050407@egenix.com> Message-ID: On Mon, 16 May 2011, M.-A. Lemburg wrote: [...] > Just to clarify: creating an sdist really only means copying > over the files from the MANIFEST into a temporary dir and then > running tar or zip on the temporary directory. You can tell > distutils to keep the temporary dir around by using the --keep-temp > option on sdist. This will reduce the operation to just running > the archiver on the temp dir again. > > The same can be done for bdist_* commands by having distutils > skip the build process. > > Since I don't see how such trivial operations can affect the > quality of the code, the discussion sounds a bit artificial :-) Perhaps a file gets left behind the second time somehow that wasn't present the first time, or vice versa (and MANIFEST includes that file). If the presence or absence of that file breaks stuff, you end up uploading a broken file. Doing your release in an automated way makes this less likely, but anybody who has maintained a complicated buildbot setup knows that this kind of thing does happen. > Note that if you want to do full testing of a release, you'll > have to download the package from PyPI and then run the tests > on the downloaded archive to be sure that things work - everything > else is just preparation for this final run :-) The upload step is interesting one from the PoV of testing because performing it implies "this is the release, come and get it". There are other such steps (tagging, sending announcements), but this is one of them. So it's nice to be as confident as possible at this point that you didn't screw it up. John From mal at egenix.com Mon May 16 21:45:33 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 16 May 2011 21:45:33 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> <4DD0F217.1050407@egenix.com> Message-ID: <4DD17EDD.3070706@egenix.com> John J Lee wrote: > On Mon, 16 May 2011, M.-A. Lemburg wrote: > [...] >> Just to clarify: creating an sdist really only means copying >> over the files from the MANIFEST into a temporary dir and then >> running tar or zip on the temporary directory. You can tell >> distutils to keep the temporary dir around by using the --keep-temp >> option on sdist. This will reduce the operation to just running >> the archiver on the temp dir again. >> >> The same can be done for bdist_* commands by having distutils >> skip the build process. >> >> Since I don't see how such trivial operations can affect the >> quality of the code, the discussion sounds a bit artificial :-) > > Perhaps a file gets left behind the second time somehow that wasn't > present the first time, or vice versa (and MANIFEST includes that file). > If the presence or absence of that file breaks stuff, you end up > uploading a broken file. > > Doing your release in an automated way makes this less likely, but > anybody who has maintained a complicated buildbot setup knows that this > kind of thing does happen. > >> Note that if you want to do full testing of a release, you'll >> have to download the package from PyPI and then run the tests >> on the downloaded archive to be sure that things work - everything >> else is just preparation for this final run :-) > > The upload step is interesting one from the PoV of testing because > performing it implies "this is the release, come and get it". There are > other such steps (tagging, sending announcements), but this is one of > them. So it's nice to be as confident as possible at this point that > you didn't screw it up. Right, but the only way to find out is by downloading what you uploaded and testing that package :-) There are lots of things that can go wrong. In my experience, most of them are user errors, e.g. you forgot to update the MANIFEST file, bump the version, remove debug settings, etc. The chance of two consecutive runs of sdist creating different archives is rather small, compared to those sources of error. That said, it's easy to get the upload command to use an already created distribution file for the upload: just add a new distutils command which sets .distribution.dist_files to what list of files you want to upload. This could also be added as an option to the distutils upload command, so that you can run: python setup.py upload --dist-files=dist/my-release-1.2.3.* -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon May 16 21:58:50 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 16 May 2011 21:58:50 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <4DD17EDD.3070706@egenix.com> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> <4DD0F217.1050407@egenix.com> <4DD17EDD.3070706@egenix.com> Message-ID: <4DD181FA.2060905@egenix.com> M.-A. Lemburg wrote: > The chance of two consecutive runs of sdist creating different > archives is rather small, compared to those sources of error. > > That said, it's easy to get the upload command to use an > already created distribution file for the upload: just > add a new distutils command which sets .distribution.dist_files > to what list of files you want to upload. > > This could also be added as an option to the distutils > upload command, so that you can run: > > python setup.py upload --dist-files=dist/my-release-1.2.3.* > Here's a trick which will do the same without having to write any new code: First run: python setup.py sdist --keep-temp Then run your tests on the created archive locally. Second run: python setup.py sdist --dry-run upload The second run won't build a new archive, it'll just upload the already created archive to PyPI. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jjl at pobox.com Mon May 16 21:58:45 2011 From: jjl at pobox.com (John J Lee) Date: Mon, 16 May 2011 20:58:45 +0100 (BST) Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <4DD17EDD.3070706@egenix.com> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> <4DD0F217.1050407@egenix.com> <4DD17EDD.3070706@egenix.com> Message-ID: On Mon, 16 May 2011, M.-A. Lemburg wrote: > Right, but the only way to find out is by downloading > what you uploaded and testing that package :-) Sure (and I do do that myself). But finding out is one thing, and not doing it in the first place is another. > There are lots of things that can go wrong. In my experience, > most of them are user errors, e.g. you forgot to update the > MANIFEST file, bump the version, remove debug settings, etc. Again, that's true. But those risks can also be reduced, with separate measures (automation). It's still true that uploading the file you tested is better than uploading a different file. > That said, it's easy to get the upload command to use an > already created distribution file for the upload: just > add a new distutils command which sets .distribution.dist_files > to what list of files you want to upload. [...] "Just" add a new distutils command? ISTR distutils isn't very open to that? IWBNI if distutils (or some replacement for it -- distutils2?) supported this out of the box. John From mal at egenix.com Mon May 16 22:07:23 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 16 May 2011 22:07:23 +0200 Subject: [Catalog-sig] PyPI's external packages In-Reply-To: References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> <4DD0F217.1050407@egenix.com> <4DD17EDD.3070706@egenix.com> Message-ID: <4DD183FB.1080900@egenix.com> John J Lee wrote: > On Mon, 16 May 2011, M.-A. Lemburg wrote: > >> That said, it's easy to get the upload command to use an >> already created distribution file for the upload: just >> add a new distutils command which sets .distribution.dist_files >> to what list of files you want to upload. > [...] > > "Just" add a new distutils command? ISTR distutils isn't very open to > that? IWBNI if distutils (or some replacement for it -- distutils2?) > supported this out of the box. Actually, extending distutils is a lot easier than many people tend to believe. Adding a new command or extending an existing one is done by subclassing from an existing command class and then registering this new class with distutils using the cmdclass keyword argument to setup(): http://docs.python.org/distutils/extending.html?highlight=cmdclass Adding an option to distutils' upload command class in Python 3.3 would work as well, of course, but takes a lot longer to become usable for package authors. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jjl at pobox.com Mon May 16 22:07:48 2011 From: jjl at pobox.com (John J Lee) Date: Mon, 16 May 2011 21:07:48 +0100 (BST) Subject: [Catalog-sig] PyPI's external packages In-Reply-To: <4DD181FA.2060905@egenix.com> References: <20110512190448.1992.1861094948.divmod.xquotient.1335@localhost.localdomain> <4DCD4C80.10803@netwok.org> <20110513171205.1992.1907640253.divmod.xquotient.1355@localhost.localdomain> <4DD0F217.1050407@egenix.com> <4DD17EDD.3070706@egenix.com> <4DD181FA.2060905@egenix.com> Message-ID: On Mon, 16 May 2011, M.-A. Lemburg wrote: [...] > Here's a trick which will do the same without having to write > any new code: > > First run: > > python setup.py sdist --keep-temp > > Then run your tests on the created archive locally. > > Second run: > > python setup.py sdist --dry-run upload > > The second run won't build a new archive, it'll just upload the > already created archive to PyPI. Nice, thanks! John From tseaver at palladion.com Fri May 20 19:57:00 2011 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 20 May 2011 13:57:00 -0400 Subject: [Catalog-sig] PyPI looks down Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 http://www.downforeveryoneorjustme.com/pypi.python.org Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3Wq2wACgkQ+gerLs4ltQ5VSwCfZ/LqfytjG2nI1UQ7uO84nd08 kUgAn29OwUSQYDlvhPEYjgtF6W/IAl2Q =QCBX -----END PGP SIGNATURE----- From chris at simplistix.co.uk Fri May 20 20:33:20 2011 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 20 May 2011 19:33:20 +0100 Subject: [Catalog-sig] PyPI looks down In-Reply-To: References: Message-ID: <4DD6B3F0.6050209@simplistix.co.uk> On 20/05/2011 18:57, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > http://www.downforeveryoneorjustme.com/pypi.python.org HA rocks: http://b.pypi.python.org/ http://c.pypi.python.org/ http://d.pypi.python.org/ http://e.pypi.python.org/ http://f.pypi.python.org/ Now, if only the tools had caught up to use these :-/ Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From monitor at jacobian.org Fri May 20 20:49:23 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Fri, 20 May 2011 13:49:23 -0500 Subject: [Catalog-sig] [monit] pypi.python.org - Connection succeeded Message-ID: <1305917366.1@jacobian.org> Connection succeeded Service pypi.python.org Date: Fri, 20 May 2011 13:49:23 -0500 Action: alert Host: jacobian.org Description: connection succeeded to INET[pypi.python.org:80] via TCP Your faithful employee, monit From carl at oddbird.net Fri May 20 21:03:28 2011 From: carl at oddbird.net (Carl Meyer) Date: Fri, 20 May 2011 14:03:28 -0500 Subject: [Catalog-sig] PyPI looks down In-Reply-To: <4DD6B3F0.6050209@simplistix.co.uk> References: <4DD6B3F0.6050209@simplistix.co.uk> Message-ID: <4DD6BB00.6000808@oddbird.net> On 05/20/2011 01:33 PM, Chris Withers wrote: > HA rocks: > > http://b.pypi.python.org/ > http://c.pypi.python.org/ > http://d.pypi.python.org/ > http://e.pypi.python.org/ > http://f.pypi.python.org/ > > Now, if only the tools had caught up to use these :-/ At least one has ;-) http://www.pip-installer.org/en/latest/index.html#mirror-support Carl From tjreedy at udel.edu Fri May 20 21:14:58 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 20 May 2011 15:14:58 -0400 Subject: [Catalog-sig] PyPI looks down In-Reply-To: References: Message-ID: On 5/20/2011 1:57 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > http://www.downforeveryoneorjustme.com/pypi.python.org Yet another cookie dispenser:-(. I wonder how they expect to monetize cookie info. But yes, useful if one can remember. -- Terry Jan Reedy From monitor at jacobian.org Fri May 20 19:44:53 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Fri, 20 May 2011 12:44:53 -0500 Subject: [Catalog-sig] [monit] pypi.python.org - Connection failed Message-ID: <1305913496.1@jacobian.org> Connection failed Service pypi.python.org Date: Fri, 20 May 2011 12:44:53 -0500 Action: alert Host: jacobian.org Description: failed protocol test [HTTP] at INET[pypi.python.org:80] via TCP Your faithful employee, monit From chris at simplistix.co.uk Sat May 21 10:00:22 2011 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 21 May 2011 09:00:22 +0100 Subject: [Catalog-sig] PyPI looks down In-Reply-To: <4DD6BB00.6000808@oddbird.net> References: <4DD6B3F0.6050209@simplistix.co.uk> <4DD6BB00.6000808@oddbird.net> Message-ID: <4DD77116.3060009@simplistix.co.uk> On 20/05/2011 20:03, Carl Meyer wrote: > >> Now, if only the tools had caught up to use these :-/ > > At least one has ;-) > > http://www.pip-installer.org/en/latest/index.html#mirror-support It install packages with C extensions yet? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From carl at oddbird.net Sat May 21 16:42:41 2011 From: carl at oddbird.net (Carl Meyer) Date: Sat, 21 May 2011 09:42:41 -0500 Subject: [Catalog-sig] PyPI looks down In-Reply-To: <4DD77116.3060009@simplistix.co.uk> References: <4DD6B3F0.6050209@simplistix.co.uk> <4DD6BB00.6000808@oddbird.net> <4DD77116.3060009@simplistix.co.uk> Message-ID: <4DD7CF61.1040204@oddbird.net> On 05/21/2011 03:00 AM, Chris Withers wrote: > On 20/05/2011 20:03, Carl Meyer wrote: >> >>> Now, if only the tools had caught up to use these :-/ >> >> At least one has ;-) >> >> http://www.pip-installer.org/en/latest/index.html#mirror-support > > It install packages with C extensions yet? Sure, I do it every day. You just need a compiler. Carl From mal at egenix.com Sat May 21 20:58:54 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 21 May 2011 20:58:54 +0200 Subject: [Catalog-sig] PyPI looks down In-Reply-To: <4DD7CF61.1040204@oddbird.net> References: <4DD6B3F0.6050209@simplistix.co.uk> <4DD6BB00.6000808@oddbird.net> <4DD77116.3060009@simplistix.co.uk> <4DD7CF61.1040204@oddbird.net> Message-ID: <4DD80B6E.6060105@egenix.com> Carl Meyer wrote: > On 05/21/2011 03:00 AM, Chris Withers wrote: >> On 20/05/2011 20:03, Carl Meyer wrote: >>> >>>> Now, if only the tools had caught up to use these :-/ >>> >>> At least one has ;-) >>> >>> http://www.pip-installer.org/en/latest/index.html#mirror-support >> >> It install packages with C extensions yet? > > Sure, I do it every day. You just need a compiler. ... and all the external dependencies such a package may have. pip really needs to support a binary format to be more user friendly in this respect in order to completely replace easy_install and make use of all the eggs on PyPI. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-06-20: EuroPython 2011, Florence, Italy 30 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From carl at oddbird.net Sun May 22 17:13:54 2011 From: carl at oddbird.net (Carl Meyer) Date: Sun, 22 May 2011 10:13:54 -0500 Subject: [Catalog-sig] PyPI looks down In-Reply-To: <4DD80B6E.6060105@egenix.com> References: <4DD6B3F0.6050209@simplistix.co.uk> <4DD6BB00.6000808@oddbird.net> <4DD77116.3060009@simplistix.co.uk> <4DD7CF61.1040204@oddbird.net> <4DD80B6E.6060105@egenix.com> Message-ID: <4DD92832.9020808@oddbird.net> On 05/21/2011 01:58 PM, M.-A. Lemburg wrote: > Carl Meyer wrote: >> On 05/21/2011 03:00 AM, Chris Withers wrote: >>> It install packages with C extensions yet? >> >> Sure, I do it every day. You just need a compiler. > > ... and all the external dependencies such a package may have. > > pip really needs to support a binary format to be more user > friendly in this respect in order to completely replace > easy_install and make use of all the eggs on PyPI. I would like to see a resolution to http://bugs.python.org/issue8654 first; at the moment, I consider the use of binary packages on PyPI broken (particularly for Python 3, which pip now supports), because the finding mechanism can easily download an egg that doesn't work on your build of Python. If that issue were resolved and the binary package format reflected all relevant ABI factors, I'd have to check with the other pip maintainers, but as far as I'm concerned it would be a "patches welcome" situation (provided the patch always unzipped the egg on installation, and made it possible to opt out of using binary packages). Carl From mal at egenix.com Thu May 26 22:11:05 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 26 May 2011 22:11:05 +0200 Subject: [Catalog-sig] PyPI looks down In-Reply-To: <4DD92832.9020808@oddbird.net> References: <4DD6B3F0.6050209@simplistix.co.uk> <4DD6BB00.6000808@oddbird.net> <4DD77116.3060009@simplistix.co.uk> <4DD7CF61.1040204@oddbird.net> <4DD80B6E.6060105@egenix.com> <4DD92832.9020808@oddbird.net> Message-ID: <4DDEB3D9.3080908@egenix.com> Carl Meyer wrote: > On 05/21/2011 01:58 PM, M.-A. Lemburg wrote: >> Carl Meyer wrote: >>> On 05/21/2011 03:00 AM, Chris Withers wrote: >>>> It install packages with C extensions yet? >>> >>> Sure, I do it every day. You just need a compiler. >> >> ... and all the external dependencies such a package may have. >> >> pip really needs to support a binary format to be more user >> friendly in this respect in order to completely replace >> easy_install and make use of all the eggs on PyPI. > > I would like to see a resolution to http://bugs.python.org/issue8654 > first; at the moment, I consider the use of binary packages on PyPI > broken (particularly for Python 3, which pip now supports), because the > finding mechanism can easily download an egg that doesn't work on your > build of Python. eggs mostly work for pure-Python packages and packages for Windows. The situation for other platforms is less than ideal, mostly because setuptools mainly relies on the egg file name alone, which includes a non-canoncical platform name and doesn't respect UCS2/UCS4 builds. There are essentially two ways to fix this: 1. standardize the egg filenames in a way that lets pip recognize compatible egg files 2. add a mapping of egg filenames to a platform information dictionary which pip can download to select the right egg file Option 1: "PEP 3149 - ABI version tagged .so files" could help with this option, since it addresses the various Python build options. We'd then only have to define a list or function which allows checking the platform string embedded in the filename against the platform running pip. Such a function would be useful for other things as well, especially on platforms that often change the platform name like e.g. FreeBSD or Mac OS X and Solaris which add universal builds to make things even more interesting. Such a function would also be useful for other applications, e.g. ones that try to determine plugin compatibility or want to switch implementations based on the platform, but not necessarily depending on the platforms OS version or 32/64-bit variant. The good news is that it's easy to come up with egg filenames that easy_install does not recognize, so both egg filename schemes could live side by side for a while. Option 2: This would allow more freedom and is easier to extend in the future, but also requires more work, both on the PyPI side, the distutils side and the client tools side. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 26 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-05-23: Released eGenix mx Base 3.2.0 http://python.egenix.com/ 2011-05-25: Released mxODBC 3.1.1 http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 25 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/