From robert.kern at gmail.com Sun Jan 1 14:26:24 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 01 Jan 2012 13:26:24 +0000 Subject: [Catalog-sig] bunch of spam packages In-Reply-To: <4EFEF228.8040907@simplistix.co.uk> References: <4EFEF228.8040907@simplistix.co.uk> Message-ID: On 12/31/11 11:29 AM, Chris Withers wrote: > http://pypi.python.org/pypi/girlfriends/1.0 > http://pypi.python.org/pypi/house/0.9 > http://pypi.python.org/pypi/hardwork/0.9 > http://pypi.python.org/pypi/car/0.9 > > ...all spamvertising the same website. It appears to me to be an honest attempt at humor[1] using the setuptools dependency mechanism rather than spam per se. Using Google Translate on the blog, it looks plausibly like a legitimate Chinese Python blogger. [1] Albeit lame and sexist to my American ears. Perhaps it plays better in China. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From chris at simplistix.co.uk Sun Jan 1 18:37:35 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 01 Jan 2012 17:37:35 +0000 Subject: [Catalog-sig] bunch of spam packages In-Reply-To: References: <4EFEF228.8040907@simplistix.co.uk> Message-ID: <4F0099DF.2060807@simplistix.co.uk> On 01/01/2012 13:26, Robert Kern wrote: > On 12/31/11 11:29 AM, Chris Withers wrote: >> http://pypi.python.org/pypi/girlfriends/1.0 >> http://pypi.python.org/pypi/house/0.9 >> http://pypi.python.org/pypi/hardwork/0.9 >> http://pypi.python.org/pypi/car/0.9 >> >> ...all spamvertising the same website. > > It appears to me to be an honest attempt at humor[1] using the > setuptools dependency mechanism rather than spam per se. Using Google > Translate on the blog, it looks plausibly like a legitimate Chinese > Python blogger. > > [1] Albeit lame and sexist to my American ears. Perhaps it plays better > in China. They're junk, and should be removed, along with all the .nested list printers I have to say, I like the idea of requiring moderation for your first package... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From robert.kern at gmail.com Sun Jan 1 19:35:44 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 01 Jan 2012 18:35:44 +0000 Subject: [Catalog-sig] bunch of spam packages In-Reply-To: <4F0099DF.2060807@simplistix.co.uk> References: <4EFEF228.8040907@simplistix.co.uk> <4F0099DF.2060807@simplistix.co.uk> Message-ID: On 1/1/12 5:37 PM, Chris Withers wrote: > On 01/01/2012 13:26, Robert Kern wrote: >> On 12/31/11 11:29 AM, Chris Withers wrote: >>> http://pypi.python.org/pypi/girlfriends/1.0 >>> http://pypi.python.org/pypi/house/0.9 >>> http://pypi.python.org/pypi/hardwork/0.9 >>> http://pypi.python.org/pypi/car/0.9 >>> >>> ...all spamvertising the same website. >> >> It appears to me to be an honest attempt at humor[1] using the >> setuptools dependency mechanism rather than spam per se. Using Google >> Translate on the blog, it looks plausibly like a legitimate Chinese >> Python blogger. >> >> [1] Albeit lame and sexist to my American ears. Perhaps it plays better >> in China. > > They're junk, and should be removed, I don't disagree per se[1], but I do think that in this case an email to the author (or blog comment) first would be a better response than silently removing the packages. [1] Though to be fair, we would also have to remove antigravity, which I didn't want to do before this set of joke package. Alas, there is no policy that allows antigravity while forbidding girlfriend except perhaps taste. http://pypi.python.org/pypi/antigravity/ > along with all the .nested list printers Similarly, I think that something along the lines of a single, mass-Bcc'ed email to the authors on record would be better than silently removing the packages. If only we could borrow Guido's time machine to prevent the author of _Head First Python_ from using this as an exercise... > I have to say, I like the idea of requiring moderation for your first package... Except for the nested list printers, there hasn't been a significant problem that moderation would solve, in my opinion. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Jan 2 16:36:50 2012 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 02 Jan 2012 15:36:50 +0000 Subject: [Catalog-sig] bunch of spam packages In-Reply-To: <4EFEF228.8040907@simplistix.co.uk> References: <4EFEF228.8040907@simplistix.co.uk> Message-ID: On 12/31/11 11:29 AM, Chris Withers wrote: > http://pypi.python.org/pypi/girlfriends/1.0 > http://pypi.python.org/pypi/house/0.9 > http://pypi.python.org/pypi/hardwork/0.9 > http://pypi.python.org/pypi/car/0.9 > > ...all spamvertising the same website. FWIW, the author has removed these packages. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aclark at aclark.net Tue Jan 3 03:15:27 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 02 Jan 2012 21:15:27 -0500 Subject: [Catalog-sig] Introducing: pythonpackages.com Message-ID: Hi, For the past two months I've been developing a service for developers and consumers of Python packages: - http://pythonpackages.com (And as folks who develop, discuss, and maintain the Python Package Index, this service may especially be of interest to you!) Recently, we've added the ability for folks to login with their github account. This changes the landscape of the site from anonymous to personal, and with this change the creation of "profile pages" is now possible, e.g.: - http://pythonpackages.com/user/nutjob4life We've seen a fair amount of activity on the site by anonymous users, and with this change I envision a fair amount of github user activity. Toward that end, I would appreciate it if you would take a minute to login with your github account here (requires read only access) and give the service a try: - http://pythonpackages.com/login You can also browse recent package activity (via the PyPI API) here: - http://pythonpackages.com/activity Discuss packages here: - http://pythonpackages.com/discuss And read our site documentation here: - http://pythonpackages-docs.readthedocs.org Lastly, if you have any feedback please feel free to leave a comment here: - http://pythonpackages.com/about Or open a ticket here: - https://bitbucket.org/pythonpackages/pythonpackages.com/issues/new Thanks, Alex -- Alex Clark ? http://pythonpackages.com From richard at python.org Tue Jan 3 03:42:07 2012 From: richard at python.org (Richard Jones) Date: Tue, 3 Jan 2012 13:42:07 +1100 Subject: [Catalog-sig] Introducing: pythonpackages.com In-Reply-To: References: Message-ID: On 3 January 2012 13:15, Alex Clark wrote: > For the past two months I've been developing a service for developers and > consumers of Python packages: > > - http://pythonpackages.com Awesome! Richard From chris at simplistix.co.uk Fri Jan 13 10:26:25 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 13 Jan 2012 09:26:25 +0000 Subject: [Catalog-sig] pypi and python.org down Message-ID: <4F0FF8C1.9050908@simplistix.co.uk> ...can anyone see what's going on and fix it? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From noah at coderanger.net Fri Jan 13 10:29:52 2012 From: noah at coderanger.net (Noah Kantrowitz) Date: Fri, 13 Jan 2012 01:29:52 -0800 Subject: [Catalog-sig] pypi and python.org down In-Reply-To: <4F0FF8C1.9050908@simplistix.co.uk> References: <4F0FF8C1.9050908@simplistix.co.uk> Message-ID: <3BA6AB17-4AA3-483E-820B-B05C6469B0D7@coderanger.net> Investigating now, thanks for the report! --Noah On Jan 13, 2012, at 1:26 AM, Chris Withers wrote: > ...can anyone see what's going on and fix it? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Fri Jan 13 11:18:24 2012 From: noah at coderanger.net (Noah Kantrowitz) Date: Fri, 13 Jan 2012 02:18:24 -0800 Subject: [Catalog-sig] pypi and python.org down In-Reply-To: <3BA6AB17-4AA3-483E-820B-B05C6469B0D7@coderanger.net> References: <4F0FF8C1.9050908@simplistix.co.uk> <3BA6AB17-4AA3-483E-820B-B05C6469B0D7@coderanger.net> Message-ID: <688EF6C1-B318-4BE1-A512-D12CFDA808C7@coderanger.net> Martin and XS4ALL have resolved the outage. Both sites should be back up and running. --Noah On Jan 13, 2012, at 1:29 AM, Noah Kantrowitz wrote: > Investigating now, thanks for the report! > > --Noah > > On Jan 13, 2012, at 1:26 AM, Chris Withers wrote: > >> ...can anyone see what's going on and fix it? >> >> Chris >> >> -- >> Simplistix - Content Management, Batch Processing & Python Consulting >> - http://www.simplistix.co.uk >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From martin at v.loewis.de Fri Jan 13 11:19:26 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 13 Jan 2012 11:19:26 +0100 Subject: [Catalog-sig] [Pydotorg] pypi and python.org down In-Reply-To: <4F0FF8C1.9050908@simplistix.co.uk> References: <4F0FF8C1.9050908@simplistix.co.uk> Message-ID: <4F10052E.90908@v.loewis.de> Am 13.01.2012 10:26, schrieb Chris Withers: > ...can anyone see what's going on and fix it? The machine was stuck, so I power-cycled it. Regards, Martin From aclark at aclark.net Sun Jan 22 18:04:10 2012 From: aclark at aclark.net (Alex Clark) Date: Sun, 22 Jan 2012 12:04:10 -0500 Subject: [Catalog-sig] New pythonpackages.com service coming soon Message-ID: Folks, I have created a new service aimed at making it easier to release Python packages to PyPI. The primary user is currently: me. And to date, I have only released a single package with it: Pillow (well, in fact I really only tested a portion of the release process with Pillow). It works like this: - I have created a "user" `pythonpackages` on PyPI - I have uploaded an ssh key [1]. - I have added `pythonpackages` as a maintainer of `Pillow`. - You can imagine the rest (and if you can't, it's a secret for now.) Now, I read the TOS very carefully before creating the `pythonpackages` "user". And there was nothing in it to indicate this action is anything other than "fair use". But I want to bring it to the attention of the PyPI maintainers now, in the event the service becomes popular later (I know at least I am planning to use it quite a bit. And we have ~70 beta users signed up to begin testing.) The bottom line is: there is now a "user" on the PyPI called `pythonpackages` that is in fact not a user, but a website (pythonpackages.com). By adding the "user" `pythonpackages` as a Maintainer to your package, you will be able to use the pythonpackages.com service to automate your release process in some exciting capacity, to be revealed soon. This is just one aspect of the service I am building, but it is an important milestone that I wanted to share (for obvious reasons). I welcome any comments/questions/concerns. It is my sincere hope that at the most, I am not offending anyone with my actions and at the least, I am not violating any terms or conditions that I don't know about. Sincerely, Alex Clark [1] I am using pypissh, http://pythonpackages.com/info/pypissh (many thanks to Martin von L?wis for this). -- Alex Clark ? http://pythonpackages.com From ziade.tarek at gmail.com Sun Jan 22 18:35:52 2012 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 22 Jan 2012 09:35:52 -0800 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: Missed the reply all ---------- Forwarded message ---------- From: "Tarek Ziad?" Date: Jan 22, 2012 9:35 AM Subject: Re: [Catalog-sig] New pythonpackages.com service coming soon To: "Alex Clark" The only concern I have is securiy. if someone breaks your server it can create havoc for those packages on PyPI. Maybe there's a way to make this more secure, like making session based authorization ? Or that's what you planned maybe ? Otherwise cool idea Cheers Tarek On Jan 22, 2012 9:04 AM, "Alex Clark" wrote: > Folks, > > I have created a new service aimed at making it easier to release Python > packages to PyPI. The primary user is currently: me. And to date, I have > only released a single package with it: Pillow (well, in fact I really only > tested a portion of the release process with Pillow). > > It works like this: > > - I have created a "user" `pythonpackages` on PyPI > - I have uploaded an ssh key [1]. > - I have added `pythonpackages` as a maintainer of `Pillow`. > - You can imagine the rest (and if you can't, it's a secret for now.) > > Now, I read the TOS very carefully before creating the `pythonpackages` > "user". And there was nothing in it to indicate this action is anything > other than "fair use". But I want to bring it to the attention of the PyPI > maintainers now, in the event the service becomes popular later (I know at > least I am planning to use it quite a bit. And we have ~70 beta users > signed up to begin testing.) > > The bottom line is: there is now a "user" on the PyPI called > `pythonpackages` that is in fact not a user, but a website ( > pythonpackages.com). By adding the "user" `pythonpackages` as a > Maintainer to your package, you will be able to use the pythonpackages.comservice to automate your release process in some exciting capacity, to be > revealed soon. This is just one aspect of the service I am building, but it > is an important milestone that I wanted to share (for obvious reasons). > > I welcome any comments/questions/concerns. It is my sincere hope that at > the most, I am not offending anyone with my actions and at the least, I am > not violating any terms or conditions that I don't know about. > > Sincerely, > > > Alex Clark > > > [1] I am using pypissh, http://pythonpackages.com/**info/pypissh(many thanks to Martin von L?wis for this). > > > -- > Alex Clark ? http://pythonpackages.com > > ______________________________**_________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/**mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Sun Jan 22 21:57:21 2012 From: aclark at aclark.net (Alex Clark) Date: Sun, 22 Jan 2012 15:57:21 -0500 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: On 1/22/12 12:35 PM, Tarek Ziad? wrote: > Missed the reply all > > ---------- Forwarded message ---------- > From: "Tarek Ziad?" > > Date: Jan 22, 2012 9:35 AM > Subject: Re: [Catalog-sig] New pythonpackages.com > service coming soon > To: "Alex Clark" > > > The only concern I have is securiy. if someone breaks your server it can > create havoc for those packages on PyPI. To address this, I'll most likely move the site to heroku where it will run on lxc-contained [1], ephemeral instances with configuration stored in the environment only [2]. > Maybe there's a way to make > this more secure, like making session based authorization ? Or that's > what you planned maybe ? I'm not sure what you mean, but I'm certainly planning lots of things for the future, assuming things go well. WRT to sessions the app currently uses Pyramid's auth_tkt policy, which configures a session for anyone that authorizes the app on github.com. > Otherwise cool idea Thanks Alex [1] http://lxc.sourceforge.net/ [2] http://devcenter.heroku.com/articles/config-vars#an_example > > Cheers > Tarek > > On Jan 22, 2012 9:04 AM, "Alex Clark" > wrote: > > Folks, > > I have created a new service aimed at making it easier to release > Python packages to PyPI. The primary user is currently: me. And to > date, I have only released a single package with it: Pillow (well, > in fact I really only tested a portion of the release process with > Pillow). > > It works like this: > > - I have created a "user" `pythonpackages` on PyPI > - I have uploaded an ssh key [1]. > - I have added `pythonpackages` as a maintainer of `Pillow`. > - You can imagine the rest (and if you can't, it's a secret for now.) > > Now, I read the TOS very carefully before creating the > `pythonpackages` "user". And there was nothing in it to indicate > this action is anything other than "fair use". But I want to bring > it to the attention of the PyPI maintainers now, in the event the > service becomes popular later (I know at least I am planning to use > it quite a bit. And we have ~70 beta users signed up to begin testing.) > > The bottom line is: there is now a "user" on the PyPI called > `pythonpackages` that is in fact not a user, but a website > (pythonpackages.com ). By adding the > "user" `pythonpackages` as a Maintainer to your package, you will be > able to use the pythonpackages.com > service to automate your release process in some exciting capacity, > to be revealed soon. This is just one aspect of the service I am > building, but it is an important milestone that I wanted to share > (for obvious reasons). > > I welcome any comments/questions/concerns. It is my sincere hope > that at the most, I am not offending anyone with my actions and at > the least, I am not violating any terms or conditions that I don't > know about. > > Sincerely, > > > Alex Clark > > > [1] I am using pypissh, http://pythonpackages.com/__info/pypissh > (many thanks to Martin von > L?wis for this). > > > -- > Alex Clark ? http://pythonpackages.com > > _________________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/__mailman/listinfo/catalog-sig > > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Alex Clark ? http://pythonpackages.com From richard at python.org Sun Jan 22 23:45:21 2012 From: richard at python.org (Richard Jones) Date: Mon, 23 Jan 2012 09:45:21 +1100 Subject: [Catalog-sig] New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: On 23 January 2012 04:04, Alex Clark wrote: > - I have created a "user" `pythonpackages` on PyPI > - I have uploaded an ssh key [1]. > - I have added `pythonpackages` as a maintainer of `Pillow`. > - You can imagine the rest (and if you can't, it's a secret for now.) > > Now, I read the TOS very carefully before creating the `pythonpackages` > "user". And there was nothing in it to indicate this action is anything > other than "fair use". But I want to bring it to the attention of the PyPI > maintainers now, in the event the service becomes popular later (I know at > least I am planning to use it quite a bit. And we have ~70 beta users signed > up to begin testing.) My initial only concern is that the registering and uploading of packages to the index might become too anonymous. We are frequently called upon to identify the owners of packages (for a variety of reasons: ownership disputes, transfer of ownership, reclamation of zombies, that sort of thing). Currently a person must be registered with PyPI an listed as an owner/maintainer to be able to register package releases and upload files for a package. Even if we required a non-pythonpackages user to be listed against a package that association could become stale (the person listed in PyPI could have no longer have anything to do with the package.) Richard From aclark at aclark.net Mon Jan 23 00:20:47 2012 From: aclark at aclark.net (Alex Clark) Date: Sun, 22 Jan 2012 18:20:47 -0500 Subject: [Catalog-sig] New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: <4F1C99CF.4000509@aclark.net> On 1/22/12 5:45 PM, Richard Jones wrote: > On 23 January 2012 04:04, Alex Clark wrote: >> - I have created a "user" `pythonpackages` on PyPI >> - I have uploaded an ssh key [1]. >> - I have added `pythonpackages` as a maintainer of `Pillow`. >> - You can imagine the rest (and if you can't, it's a secret for now.) >> >> Now, I read the TOS very carefully before creating the `pythonpackages` >> "user". And there was nothing in it to indicate this action is anything >> other than "fair use". But I want to bring it to the attention of the PyPI >> maintainers now, in the event the service becomes popular later (I know at >> least I am planning to use it quite a bit. And we have ~70 beta users signed >> up to begin testing.) > > My initial only concern is that the registering and uploading of > packages to the index might become too anonymous. > > We are frequently called upon to identify the owners of packages (for > a variety of reasons: ownership disputes, transfer of ownership, > reclamation of zombies, that sort of thing). > > Currently a person must be registered with PyPI an listed as an > owner/maintainer to be able to register package releases and upload > files for a package. Even if we required a non-pythonpackages user to > be listed against a package that association could become stale (the > person listed in PyPI could have no longer have anything to do with > the package.) That shouldn't be a concern here because anyone that wants to use the service (currently) must manually assign the Maintainer role to the `pythonpackages` user for their package(s). We (currently) have no plans to register any new packages with the `pythonpackages` user. Our plans could change in the future, but at present this is a small, cautious step towards release automation. And in general, the service is not intended to anonymize releases; rather, the initial set of uploads will be coming from folks that meet the following criteria: - Github user - PyPI user with at least one released package - pythonpackages.com beta member Alex > > > Richard -- Alex Clark ? http://pythonpackages.com From richard at python.org Mon Jan 23 00:34:31 2012 From: richard at python.org (Richard Jones) Date: Mon, 23 Jan 2012 10:34:31 +1100 Subject: [Catalog-sig] New pythonpackages.com service coming soon In-Reply-To: <4F1C99CF.4000509@aclark.net> References: <4F1C99CF.4000509@aclark.net> Message-ID: On 23 January 2012 10:20, Alex Clark wrote: > On 1/22/12 5:45 PM, Richard Jones wrote: >> >> On 23 January 2012 04:04, Alex Clark ?wrote: >>> >>> - I have created a "user" `pythonpackages` on PyPI >>> - I have uploaded an ssh key [1]. >>> - I have added `pythonpackages` as a maintainer of `Pillow`. >>> - You can imagine the rest (and if you can't, it's a secret for now.) >>> >>> Now, I read the TOS very carefully before creating the `pythonpackages` >>> "user". And there was nothing in it to indicate this action is anything >>> other than "fair use". But I want to bring it to the attention of the >>> PyPI >>> maintainers now, in the event the service becomes popular later (I know >>> at >>> least I am planning to use it quite a bit. And we have ~70 beta users >>> signed >>> up to begin testing.) >> >> >> My initial only concern is that the registering and uploading of >> packages to the index might become too anonymous. >> >> We are frequently called upon to identify the owners of packages (for >> a variety of reasons: ownership disputes, transfer of ownership, >> reclamation of zombies, that sort of thing). >> >> Currently a person must be registered with PyPI an listed as an >> owner/maintainer to be able to register package releases and upload >> files for a package. Even if we required a non-pythonpackages user to >> be listed against a package that association could become stale (the >> person listed in PyPI could have no longer have anything to do with >> the package.) > > That shouldn't be a concern here because anyone that wants to use the > service (currently) must manually assign the Maintainer role to the > `pythonpackages` user for their package(s). We (currently) have no plans to > register any new packages with the `pythonpackages` user. Our plans could > change in the future, but at present this is a small, cautious step towards > release automation. My concern was that in the longer term this could happen: 1. user registers package on pypi (and is thus owner) 2. user assigns pythonpackages as co-maintainer 3. user and others in package project use pythonpackages to submit new releases (possibly automa[tg]ically using mechanisms set up by the user from step #1 that they aren't fully aware of) 4. time passes and user from step #1 no longer participates in project 5. there is now effectively no useful human assigned to the package on pypi, yet releases may still happen As I said before, we frequently get requests for ownership reassignment. In this case we the original owner is not contactable / helpful (this happens a bit.) We can see there's more recent releases but we don't know who is performing them. We are now in a bind, or have to spend a bunch more effort to figure out what's going on - and we're already somewhat stretched (for two volunteers) with the current setup. Richard From ziade.tarek at gmail.com Mon Jan 23 00:37:24 2012 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 22 Jan 2012 15:37:24 -0800 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: On Sun, Jan 22, 2012 at 12:57 P > > Maybe there's a way to make >> this more secure, like making session based authorization ? Or that's >> what you planned maybe ? >> > > I'm not sure what you mean, but I'm certainly planning lots of things for > the future, assuming things go well. WRT to sessions the app currently uses > Pyramid's auth_tkt policy, which configures a session for anyone that > authorizes the app on github.com. > I meant giving a temporary access to my PyPI packages from within your application when performing tasks, not a complete & permanent one where you application could perform unwanted tasks at PyPI if the server gets hacked. I am not sure how this could be done practically speaking, it depends on the client UI. Cheers Tarek > > Otherwise cool idea >> > > Thanks > > > Alex > > [1] http://lxc.sourceforge.net/ > [2] http://devcenter.heroku.com/**articles/config-vars#an_**example > > > >> Cheers >> Tarek >> >> On Jan 22, 2012 9:04 AM, "Alex Clark" > > wrote: >> >> Folks, >> >> I have created a new service aimed at making it easier to release >> Python packages to PyPI. The primary user is currently: me. And to >> date, I have only released a single package with it: Pillow (well, >> in fact I really only tested a portion of the release process with >> Pillow). >> >> It works like this: >> >> - I have created a "user" `pythonpackages` on PyPI >> - I have uploaded an ssh key [1]. >> - I have added `pythonpackages` as a maintainer of `Pillow`. >> - You can imagine the rest (and if you can't, it's a secret for now.) >> >> Now, I read the TOS very carefully before creating the >> `pythonpackages` "user". And there was nothing in it to indicate >> this action is anything other than "fair use". But I want to bring >> it to the attention of the PyPI maintainers now, in the event the >> service becomes popular later (I know at least I am planning to use >> it quite a bit. And we have ~70 beta users signed up to begin testing.) >> >> The bottom line is: there is now a "user" on the PyPI called >> `pythonpackages` that is in fact not a user, but a website >> (pythonpackages.com ). By adding the >> >> "user" `pythonpackages` as a Maintainer to your package, you will be >> able to use the pythonpackages.com >> >> service to automate your release process in some exciting capacity, >> to be revealed soon. This is just one aspect of the service I am >> building, but it is an important milestone that I wanted to share >> (for obvious reasons). >> >> I welcome any comments/questions/concerns. It is my sincere hope >> that at the most, I am not offending anyone with my actions and at >> the least, I am not violating any terms or conditions that I don't >> know about. >> >> Sincerely, >> >> >> Alex Clark >> >> >> [1] I am using pypissh, http://pythonpackages.com/__**info/pypissh >> >> > >> (many thanks to Martin von >> L?wis for this). >> >> >> -- >> Alex Clark ? http://pythonpackages.com >> >> ______________________________**___________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/__**mailman/listinfo/catalog-sig >> >> > >> >> >> >> >> ______________________________**_________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/**mailman/listinfo/catalog-sig >> > > > -- > Alex Clark ? http://pythonpackages.com > > ______________________________**_________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/**mailman/listinfo/catalog-sig > -- Tarek Ziad? | http://ziade.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Mon Jan 23 00:49:35 2012 From: richard at python.org (Richard Jones) Date: Mon, 23 Jan 2012 10:49:35 +1100 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: On 23 January 2012 10:37, Tarek Ziad? wrote: > I meant giving a temporary access to my PyPI packages from within your > application when performing tasks, not a complete & permanent one where you > application could perform unwanted tasks at PyPI if the server gets hacked. If I understand you correctly you are talking about using a mechanism like OpenAuth? PyPI currently only provides OpenID support, not OpenAuth. I don't recall there having been a discussion about adding OpenAuth, though I certainly can't immediately think of a reason not to add it (except that someone has to do it ;-)* Richard * if there is interest I could do it during the US PyCon sprints... From ziade.tarek at gmail.com Mon Jan 23 00:53:55 2012 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 22 Jan 2012 15:53:55 -0800 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: On Sun, Jan 22, 2012 at 3:49 PM, Richard Jones wrote: > On 23 January 2012 10:37, Tarek Ziad? wrote: > > I meant giving a temporary access to my PyPI packages from within your > > application when performing tasks, not a complete & permanent one where > you > > application could perform unwanted tasks at PyPI if the server gets > hacked. > > If I understand you correctly you are talking about using a mechanism > like OpenAuth? PyPI currently only provides OpenID support, not > OpenAuth. I don't recall there having been a discussion about adding > OpenAuth, though I certainly can't immediately think of a reason not > to add it (except that someone has to do it ;-)* > Yeah for example > > Richard > > * if there is interest I could do it during the US PyCon sprints... > -- Tarek Ziad? | http://ziade.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Mon Jan 23 01:00:03 2012 From: aclark at aclark.net (Alex Clark) Date: Sun, 22 Jan 2012 19:00:03 -0500 Subject: [Catalog-sig] New pythonpackages.com service coming soon In-Reply-To: References: <4F1C99CF.4000509@aclark.net> Message-ID: On 1/22/12 6:34 PM, Richard Jones wrote: > On 23 January 2012 10:20, Alex Clark wrote: >> On 1/22/12 5:45 PM, Richard Jones wrote: >>> >>> On 23 January 2012 04:04, Alex Clark wrote: >>>> >>>> - I have created a "user" `pythonpackages` on PyPI >>>> - I have uploaded an ssh key [1]. >>>> - I have added `pythonpackages` as a maintainer of `Pillow`. >>>> - You can imagine the rest (and if you can't, it's a secret for now.) >>>> >>>> Now, I read the TOS very carefully before creating the `pythonpackages` >>>> "user". And there was nothing in it to indicate this action is anything >>>> other than "fair use". But I want to bring it to the attention of the >>>> PyPI >>>> maintainers now, in the event the service becomes popular later (I know >>>> at >>>> least I am planning to use it quite a bit. And we have ~70 beta users >>>> signed >>>> up to begin testing.) >>> >>> >>> My initial only concern is that the registering and uploading of >>> packages to the index might become too anonymous. >>> >>> We are frequently called upon to identify the owners of packages (for >>> a variety of reasons: ownership disputes, transfer of ownership, >>> reclamation of zombies, that sort of thing). >>> >>> Currently a person must be registered with PyPI an listed as an >>> owner/maintainer to be able to register package releases and upload >>> files for a package. Even if we required a non-pythonpackages user to >>> be listed against a package that association could become stale (the >>> person listed in PyPI could have no longer have anything to do with >>> the package.) >> >> That shouldn't be a concern here because anyone that wants to use the >> service (currently) must manually assign the Maintainer role to the >> `pythonpackages` user for their package(s). We (currently) have no plans to >> register any new packages with the `pythonpackages` user. Our plans could >> change in the future, but at present this is a small, cautious step towards >> release automation. > > My concern was that in the longer term this could happen: > > 1. user registers package on pypi (and is thus owner) > 2. user assigns pythonpackages as co-maintainer > 3. user and others in package project use pythonpackages to submit new > releases (possibly automa[tg]ically using mechanisms set up by the > user from step #1 that they aren't fully aware of) > 4. time passes and user from step #1 no longer participates in project > 5. there is now effectively no useful human assigned to the package on > pypi, yet releases may still happen Releases may technically still be possible via pythonpackages.com, but practically speaking they shouldn't happen because the only person able to trigger them (from pythonpackages.com) is the user that disappeared. However, you have got me thinking about a potential abuse scenario where a "legitimate" but malicious pythonpackages.com user could release any package that had `pythonpackages` as a Maintainer. This makes think that at the very least, in addition to adding the `pythonpackages` user as Maintainer, we (pythonpackages.com) must require users to identify themselves with their PyPI openid (which of course can be used for identification, but not releasing packages). That way pythonpackages.com could verify that the package being released has the right Owner, simply by checking the package metadata and reconciling it with the openid (at least in my head this sounds like it should work). > > As I said before, we frequently get requests for ownership > reassignment. In this case we the original owner is not contactable / > helpful (this happens a bit.) We can see there's more recent releases > but we don't know who is performing them. We are now in a bind, or > have to spend a bunch more effort to figure out what's going on - and > we're already somewhat stretched (for two volunteers) with the current > setup. Indeed, I definitely don't want to create more work for anyone. Alex > > > Richard -- Alex Clark ? http://pythonpackages.com From aclark at aclark.net Mon Jan 23 01:01:26 2012 From: aclark at aclark.net (Alex Clark) Date: Sun, 22 Jan 2012 19:01:26 -0500 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: On 1/22/12 6:53 PM, Tarek Ziad? wrote: > > > On Sun, Jan 22, 2012 at 3:49 PM, Richard Jones > wrote: > > On 23 January 2012 10:37, Tarek Ziad? > wrote: > > I meant giving a temporary access to my PyPI packages from within > your > > application when performing tasks, not a complete & permanent one > where you > > application could perform unwanted tasks at PyPI if the server > gets hacked. > > If I understand you correctly you are talking about using a mechanism > like OpenAuth? PyPI currently only provides OpenID support, not > OpenAuth. I don't recall there having been a discussion about adding > OpenAuth, though I certainly can't immediately think of a reason not > to add it (except that someone has to do it ;-)* > > > Yeah for example > > > > > Richard > > * if there is interest I could do it during the US PyCon sprints... OAuth would be a most welcome addition! > > > > > > > -- > Tarek Ziad? | http://ziade.org > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Alex Clark ? http://pythonpackages.com From thulka at discretelogics.com Sun Jan 22 21:11:57 2012 From: thulka at discretelogics.com (Thomas Hulka) Date: Sun, 22 Jan 2012 21:11:57 +0100 Subject: [Catalog-sig] "testime" : package name conflict Message-ID: Hello, We would like to publish an open source package on PyPI for our product line "teatime". We would like to call the package teatime, but a package with such name is already registered. The registered package is surely a funny package but not of essential value for the python community and has not been touched since submission many years ago, it appears to be more of a trial to upload some code. The issue now is that package author Etienne Robillard is not reachable under the email given here or under any email address I found on the web. Is there any cleaning procedure that removes packages from the repository if the authors email becomes invalid, the package is hardly downloaded, of obvious limited value and not maintained anymore? Cheers Thomas -- Thomas Hulka *discretelogics *Kirchengasse 5/1/1 1070 Wien Austria thulka at discretelogics.com 0650 216 39 60 -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 23 23:26:53 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 Jan 2012 23:26:53 +0100 Subject: [Catalog-sig] "testime" : package name conflict In-Reply-To: References: Message-ID: <4F1DDEAD.3000403@v.loewis.de> > We would like to publish an open source package on PyPI for our product > line "teatime". We would like to call the package teatime, but a package > with such name is already registered. The registered package is surely a > funny package but not of essential value for the python community and > has not been touched since submission many years ago, it appears to be > more of a trial to upload some code. The issue now is that package > author Etienne Robillard is not reachable under the email given here or > under any email address I found on the web. Is there any cleaning > procedure that removes packages from the repository if the authors email > becomes invalid, the package is hardly downloaded, of obvious limited > value and not maintained anymore? That the package appears to have little value to you is no reason to delete it from the package index. The policy for using package names is clearly "first-come, first served", except in extraordinary circumstances such as trademark infringement. That the author is unresponsive is more of a reason to delete the package from the index. In the past, we used that as a reason to reassign package maintenance to a new maintainer. Reusing the name for a different project entirely is a different issue. It may be that the original author comes back, says he was on vacation, and wants the reassignment reverted, in which case we should revert it. That said, please submit a support request to the PyPI issue tracker. We'll probably verify unreachability of the original user of the name also, and then reassign. Please understand that this may take a while. Regards, Martin From martin at v.loewis.de Mon Jan 23 23:30:49 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 Jan 2012 23:30:49 +0100 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: Message-ID: <4F1DDF99.8020703@v.loewis.de> > OAuth would be a most welcome addition! Can you please flesh out a specification? I'd be hesitant to add it without a *clear* commitment to using it. We added OpenID support primarily to support pythonpackages.com, only to find out that it now uses github accounts :-( I'd be angry to learn that I implemented yet another feature which is then not going to be used. Regards, Martin From donald.stufft at gmail.com Mon Jan 23 23:37:59 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 23 Jan 2012 17:37:59 -0500 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: <4F1DDF99.8020703@v.loewis.de> References: <4F1DDF99.8020703@v.loewis.de> Message-ID: <2E87603F66DF41E6B409430C0186EF91@gmail.com> Unrelated but: PyPI works as an openid provider? Is there any documentation for this? On Monday, January 23, 2012 at 5:30 PM, "Martin v. L?wis" wrote: > > OAuth would be a most welcome addition! > > > Can you please flesh out a specification? > > I'd be hesitant to add it without a *clear* commitment to using it. > > We added OpenID support primarily to support pythonpackages.com (http://pythonpackages.com), > only to find out that it now uses github accounts :-( I'd be angry > to learn that I implemented yet another feature which is then not > going to be used. > > Regards, > Martin > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Jan 24 00:12:24 2012 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 24 Jan 2012 00:12:24 +0100 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: <2E87603F66DF41E6B409430C0186EF91@gmail.com> References: <4F1DDF99.8020703@v.loewis.de> <2E87603F66DF41E6B409430C0186EF91@gmail.com> Message-ID: <4F1DE958.5000301@v.loewis.de> Am 23.01.2012 23:37, schrieb Donald Stufft: > Unrelated but: PyPI works as an openid provider? Is there any > documentation for this? Only on catalog SIG: http://mail.python.org/pipermail/catalog-sig/2011-November/004066.html You can (now) use pypi.python.org as the OpenID provider name, i.e. log in with pypi.python.org into any compliant relying party. If you want to type your full OpenID, it's http://pypi.python.org/id/ I plan to put this on the user's settings page. Regards, Martin From richard at python.org Tue Jan 24 00:44:00 2012 From: richard at python.org (Richard Jones) Date: Tue, 24 Jan 2012 10:44:00 +1100 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: <4F1DE958.5000301@v.loewis.de> References: <4F1DDF99.8020703@v.loewis.de> <2E87603F66DF41E6B409430C0186EF91@gmail.com> <4F1DE958.5000301@v.loewis.de> Message-ID: On 24 January 2012 10:12, "Martin v. L?wis" wrote: > Am 23.01.2012 23:37, schrieb Donald Stufft: >> Unrelated but: PyPI works as an openid provider? Is there any >> documentation for this? > > Only on catalog SIG: > > http://mail.python.org/pipermail/catalog-sig/2011-November/004066.html Before you ask: we were waiting for one of the couple of interested trial sites to implement the OpenID setup before announcing it more publicly. Richard From donald.stufft at gmail.com Tue Jan 24 00:47:48 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 23 Jan 2012 18:47:48 -0500 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: <4F1DDF99.8020703@v.loewis.de> <2E87603F66DF41E6B409430C0186EF91@gmail.com> <4F1DE958.5000301@v.loewis.de> Message-ID: <42CC4AAA83184CFF9DAFEA6C08D37E22@gmail.com> Well I'm interested in PyPI OpenID ;) (or OAuth, either way? OAuth would be nice in that people could give authorization to specific packages, and be more comprehensive then just a Login) On Monday, January 23, 2012 at 6:44 PM, Richard Jones wrote: > On 24 January 2012 10:12, "Martin v. L?wis" wrote: > > Am 23.01.2012 23:37, schrieb Donald Stufft: > > > Unrelated but: PyPI works as an openid provider? Is there any > > > documentation for this? > > > > > > > > > Only on catalog SIG: > > > > http://mail.python.org/pipermail/catalog-sig/2011-November/004066.html > > Before you ask: we were waiting for one of the couple of interested > trial sites to implement the OpenID setup before announcing it more > publicly. > > > Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Tue Jan 24 01:13:13 2012 From: richard at python.org (Richard Jones) Date: Tue, 24 Jan 2012 11:13:13 +1100 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: <42CC4AAA83184CFF9DAFEA6C08D37E22@gmail.com> References: <4F1DDF99.8020703@v.loewis.de> <2E87603F66DF41E6B409430C0186EF91@gmail.com> <4F1DE958.5000301@v.loewis.de> <42CC4AAA83184CFF9DAFEA6C08D37E22@gmail.com> Message-ID: On 24 January 2012 10:47, Donald Stufft wrote: > Well I'm interested in PyPI OpenID ;) (or OAuth, either way? OAuth would be > nice in that people could give authorization to specific packages, and be > more comprehensive then just a Login) Could you explain what you mean by "people could give authorization to specific packages"? Do you have a specific use-case in mind? Do you have a site that intends to use PyPI's OpenID? Richard From aclark at aclark.net Tue Jan 24 02:22:03 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 23 Jan 2012 20:22:03 -0500 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: <4F1DDF99.8020703@v.loewis.de> References: <4F1DDF99.8020703@v.loewis.de> Message-ID: Hi Martin, On 1/23/12 5:30 PM, "Martin v. L?wis" wrote: >> OAuth would be a most welcome addition! > > Can you please flesh out a specification? > > I'd be hesitant to add it without a *clear* commitment to using it. > > We added OpenID support primarily to support pythonpackages.com, > only to find out that it now uses github accounts :-( I'd be angry > to learn that I implemented yet another feature which is then not > going to be used. OpenID is still on the table, so I don't want you to get the impression that we're jumping ship for OAuth. That said, I apologize about leaving you hanging there; it was certainly not my intention (and I was unaware until now that OpenId support was added for pythonpackages.com? unless maybe you are confusing it with opencomparison.org?). In any event, yes, I can put together a specification. Things should be easier to discuss now that I have announced the details. Let me do this: - Between now and March, I'll implement OpenId support on pythonpackages.com. - That support will, initially, only be used to verify that someone with a Github account who has already signed in owns a particular package on PyPI. As that is clumsy even to describe, I suspect it will only be a stepping stone to a better approach (but I think it gets me what I want, which is to publish packages that have shared the maintainer role with `pythonpackages`. I certainly don't want any pythonpackages.com user to be able to publish any package on PyPI that has done the same.) Alex > > Regards, > Martin -- Alex Clark ? http://pythonpackages.com From donald.stufft at gmail.com Tue Jan 24 02:23:22 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 23 Jan 2012 20:23:22 -0500 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: References: <4F1DDF99.8020703@v.loewis.de> <2E87603F66DF41E6B409430C0186EF91@gmail.com> <4F1DE958.5000301@v.loewis.de> <42CC4AAA83184CFF9DAFEA6C08D37E22@gmail.com> Message-ID: <8186C32107F044EC9857D78C779B280E@gmail.com> If i'm the owner of package foo, and website bar.com wants to modify my PyPI listing, or get private information, or whatever OAuth could be used to securely grant bar.com authorization to the foo resource. And I wasn't aware of PyPI's OpenID support, but now that I know of it I believe I have some ideas for taking advantage of it yes. On Monday, January 23, 2012 at 7:13 PM, Richard Jones wrote: > On 24 January 2012 10:47, Donald Stufft wrote: > > Well I'm interested in PyPI OpenID ;) (or OAuth, either way? OAuth would be > > nice in that people could give authorization to specific packages, and be > > more comprehensive then just a Login) > > > > > Could you explain what you mean by "people could give authorization to > specific packages"? Do you have a specific use-case in mind? Do you > have a site that intends to use PyPI's OpenID? > > > Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Jan 24 02:27:38 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 23 Jan 2012 20:27:38 -0500 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: <8186C32107F044EC9857D78C779B280E@gmail.com> References: <4F1DDF99.8020703@v.loewis.de> <2E87603F66DF41E6B409430C0186EF91@gmail.com> <4F1DE958.5000301@v.loewis.de> <42CC4AAA83184CFF9DAFEA6C08D37E22@gmail.com> <8186C32107F044EC9857D78C779B280E@gmail.com> Message-ID: <2CD74BA73D074BC8858B252998CDA8FA@gmail.com> The general gist is, that the only way to grant an external service any access to your package is by either giving them your username/password, or by having a general user account for that service (similar to Alex Clark's `python packages`) user. Utilizing OAuth (beyond a basic log into external site with pypi creeds) would give a secure way for an owner to grant authorization for an external service to a resource (in this case a package). Without needing to resort to the hackish fake user accounts. On Monday, January 23, 2012 at 8:23 PM, Donald Stufft wrote: > If i'm the owner of package foo, and website bar.com (http://bar.com) wants to modify my PyPI listing, or get private information, or whatever OAuth could be used to securely grant bar.com (http://bar.com) authorization to the foo resource. > > And I wasn't aware of PyPI's OpenID support, but now that I know of it I believe I have some ideas for taking advantage of it yes. > > On Monday, January 23, 2012 at 7:13 PM, Richard Jones wrote: > > > On 24 January 2012 10:47, Donald Stufft wrote: > > > Well I'm interested in PyPI OpenID ;) (or OAuth, either way? OAuth would be > > > nice in that people could give authorization to specific packages, and be > > > more comprehensive then just a Login) > > > > > > > > > Could you explain what you mean by "people could give authorization to > > specific packages"? Do you have a specific use-case in mind? Do you > > have a site that intends to use PyPI's OpenID? > > > > > > Richard > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Jan 24 02:57:25 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Tue, 24 Jan 2012 02:57:25 +0100 Subject: [Catalog-sig] Fwd: Re: New pythonpackages.com service coming soon In-Reply-To: <8186C32107F044EC9857D78C779B280E@gmail.com> References: <4F1DDF99.8020703@v.loewis.de> <2E87603F66DF41E6B409430C0186EF91@gmail.com> <4F1DE958.5000301@v.loewis.de> <42CC4AAA83184CFF9DAFEA6C08D37E22@gmail.com> <8186C32107F044EC9857D78C779B280E@gmail.com> Message-ID: <20120124025725.Horde.W7R-OElCcOxPHhAF8Ziz3gA@webmail.df.eu> Pardon me for jumping in, but this isn't really the *specific* use-case that Richard was asking for: AFAICT, you are *not* the owner of package "foo" (that is owned by "munichlinux"), and I doubt that bar.com has any interest in PyPI (as that primarily seems to collect bar jokes, both of the "A guy walks into a bar" kind, and of the lawyer kind) Plus, there isn't really a PyPI listing of any user, nor is there much private information that anybody would want to get on PyPI (and for what little private information there is, I can't see any use case for getting it). Regards, Martin > If i'm the owner of package foo, and website bar.com wants to modify > my PyPI listing, or get private information, or whatever OAuth could > be used to securely grant bar.com authorization to the foo resource. > > And I wasn't aware of PyPI's OpenID support, but now that I know of > it I believe I have some ideas for taking advantage of it yes. > > > On Monday, January 23, 2012 at 7:13 PM, Richard Jones wrote: > >> On 24 January 2012 10:47, Donald Stufft > (mailto:donald.stufft at gmail.com)> wrote: >> > Well I'm interested in PyPI OpenID ;) (or OAuth, either way? >> OAuth would be >> > nice in that people could give authorization to specific packages, and be >> > more comprehensive then just a Login) >> > >> >> >> Could you explain what you mean by "people could give authorization to >> specific packages"? Do you have a specific use-case in mind? Do you >> have a site that intends to use PyPI's OpenID? >> >> >> Richard From tjreedy at udel.edu Tue Jan 24 05:43:13 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 23 Jan 2012 23:43:13 -0500 Subject: [Catalog-sig] "testime" : package name conflict In-Reply-To: <4F1DDEAD.3000403@v.loewis.de> References: <4F1DDEAD.3000403@v.loewis.de> Message-ID: On 1/23/2012 5:26 PM, "Martin v. L?wis" wrote: > That the package appears to have little value to you is no reason to > delete it from the package index. The policy for using package names > is clearly "first-come, first served", except in extraordinary > circumstances such as trademark infringement. > > That the author is unresponsive is more of a reason to delete the > package from the index. In the past, we used that as a reason to > reassign package maintenance to a new maintainer. In this case, the project url http://www.gthc.org/ redirects to http://tkadm30.elite-server.co.uk/ an 'Apache Test Page', and the project download page http://www.gthc.org/distfiles/teatime-1.0.2.tar.gz is not found. (Too bad, my daugher might have found some interest in whatever this was.) If this continue very long, this would seem like a good candidate for removal, freeing up the name. Perhaps we should have a simple link checker run every so often and if all project urls stay out of service for a while flag the project somehow, and eventually delete. -- Terry Jan Reedy From richard at python.org Tue Jan 24 05:52:37 2012 From: richard at python.org (Richard Jones) Date: Tue, 24 Jan 2012 15:52:37 +1100 Subject: [Catalog-sig] "testime" : package name conflict In-Reply-To: <4F1DDEAD.3000403@v.loewis.de> References: <4F1DDEAD.3000403@v.loewis.de> Message-ID: On 24 January 2012 09:26, "Martin v. L?wis" wrote: >> We would like to publish an open source package on PyPI for our product >> line "teatime". We would like to call the package teatime, but a package >> with such name is already registered. The registered package is surely a >> funny package but not of essential value for the python community and >> has not been touched since submission many years ago, it appears to be >> more of a trial to upload some code. The issue now is that package >> author Etienne Robillard is not reachable under the email given here or >> under any email address I found on the web. Is there any cleaning >> procedure that removes packages from the repository if the authors email >> becomes invalid, the package is hardly downloaded, of obvious limited >> value and not maintained anymore? > > That the package appears to have little value to you is no reason to > delete it from the package index This is my response also. There is software uploaded to the PyPI package entry. I see no reason to nuke it out of hand even if the author of the package is no longer contactable. Richard From martin at v.loewis.de Tue Jan 24 09:20:24 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Tue, 24 Jan 2012 09:20:24 +0100 Subject: [Catalog-sig] "testime" : package name conflict In-Reply-To: References: <4F1DDEAD.3000403@v.loewis.de> Message-ID: <20120124092024.Horde.63iVfNjz9kRPHmnIhfmXx8A@webmail.df.eu> > In this case, the project url http://www.gthc.org/ > redirects to http://tkadm30.elite-server.co.uk/ > an 'Apache Test Page', and the project download page > http://www.gthc.org/distfiles/teatime-1.0.2.tar.gz > is not found. (Too bad, my daugher might have found some interest in > whatever this was.) Just point your daughter to http://pypi.python.org/packages/source/t/teatime/teatime-1.0.2.tar.gz which is still available. Regards, Martin From mal at egenix.com Tue Jan 24 09:45:11 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 24 Jan 2012 09:45:11 +0100 Subject: [Catalog-sig] "testime" : package name conflict In-Reply-To: <20120124092024.Horde.63iVfNjz9kRPHmnIhfmXx8A@webmail.df.eu> References: <4F1DDEAD.3000403@v.loewis.de> <20120124092024.Horde.63iVfNjz9kRPHmnIhfmXx8A@webmail.df.eu> Message-ID: <4F1E6F97.5060503@egenix.com> martin at v.loewis.de wrote: >> In this case, the project url http://www.gthc.org/ >> redirects to http://tkadm30.elite-server.co.uk/ >> an 'Apache Test Page', and the project download page >> http://www.gthc.org/distfiles/teatime-1.0.2.tar.gz >> is not found. (Too bad, my daugher might have found some interest in whatever this was.) > > Just point your daughter to > > http://pypi.python.org/packages/source/t/teatime/teatime-1.0.2.tar.gz > > which is still available. You might also want to check out: http://wiki.python.org/moin/EtienneRobillard http://code.activestate.com/pypm/author:Etienne-Robillard/ https://bitbucket.org/erob/notmm/overview While the package information may not be up to date and Etienne appears to have some problems with his websites, it doesn't seem like he's left the Python community... and there are plenty alternative email addresses to try to contact him. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From hanno at hannosch.eu Tue Jan 24 12:03:06 2012 From: hanno at hannosch.eu (Hanno Schlichting) Date: Tue, 24 Jan 2012 12:03:06 +0100 Subject: [Catalog-sig] New MPL 2.0 license classifier Message-ID: Hi. Could we get a new classifier for the MPL 2.0 license as per https://mpl.mozilla.org/2012/01/03/announcing-mpl-2-0 and http://www.opensource.org/licenses/MPL-2.0. I think it should be: License :: OSI Approved :: Mozilla Public License 2.0 (MPL) Thanks, Hanno From tjreedy at udel.edu Tue Jan 24 23:16:42 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 Jan 2012 17:16:42 -0500 Subject: [Catalog-sig] "testime" : package name conflict In-Reply-To: <20120124092024.Horde.63iVfNjz9kRPHmnIhfmXx8A@webmail.df.eu> References: <4F1DDEAD.3000403@v.loewis.de> <20120124092024.Horde.63iVfNjz9kRPHmnIhfmXx8A@webmail.df.eu> Message-ID: On 1/24/2012 3:20 AM, martin at v.loewis.de wrote: > http://pypi.python.org/packages/source/t/teatime/teatime-1.0.2.tar.gz I did not notice that the hidden url was different from the 'download' url. Looking at it, no, no reason to arbitrarily delete it, even if slightly strange. -- Terry Jan Reedy From jcea at jcea.es Wed Jan 25 17:48:03 2012 From: jcea at jcea.es (Jesus Cea) Date: Wed, 25 Jan 2012 17:48:03 +0100 Subject: [Catalog-sig] Classifier: Python 3.3 Message-ID: <4F203243.4080200@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I test my code against current 3.3 head, and I would like to mark these releases as "3.3 ready", but PYPI doesn't accept it :-). - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTyAyQ5lgi5GaxT1NAQLBIAP/TCa1bsxuVe4kL08FBrGsvK1YFMpxkGLU zwYH9NgBtH80QZx+s5gglDOuOI6TvK0F7LYdTYfCn+i+KEytwYV92QrVYJZ7MRhb Sv1Ou7g3SlXHgWGeoiwA7at3qnIeesvlzn6dZxM9pK3ga69j8V8MKnsO9OHQWyEG gHlLWNsRcjg= =DETv -----END PGP SIGNATURE----- From mikko at redinnovation.com Sun Jan 29 16:23:59 2012 From: mikko at redinnovation.com (miohtama) Date: Sun, 29 Jan 2012 07:23:59 -0800 (PST) Subject: [Catalog-sig] Distribute and endless loop trying to upgrade system-wide Distribute package in Buildout Message-ID: <1327850639567-4348358.post@n6.nabble.com> Buildout uses Distribute internally, in locally installed egg. However, due to a bug in Distribute, Buildout cannot update this egg automatically. This is because Distribute egg upgrade procedure always refers to system-wide Distribute installation, not the local one. Also, instead of proper error message or handling, the buildout goes to eternal loop. The bug have been identified here in distribute_setup.py: http://stackoverflow.com/q/5818100/315168 This bug is very common with Plone, as eventually you need to run buildout and if buildout thinks it needs to upgrade Distribute it will fail. What / who / how should be contacted to get a fix / workaround for this bug? Where to file a bug? The thread on Plone core developers list: http://plone.293351.n2.nabble.com/Buildout-and-fearsome-system-wide-Distribute-upgrade-loop-solution-available-tp7234343p7234343.html -- View this message in context: http://python.6.n6.nabble.com/Distribute-and-endless-loop-trying-to-upgrade-system-wide-Distribute-package-in-Buildout-tp4348358p4348358.html Sent from the Python - catalog-sig mailing list archive at Nabble.com. From r1chardj0n3s at gmail.com Mon Jan 30 00:47:37 2012 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Mon, 30 Jan 2012 10:47:37 +1100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole Message-ID: Hi catalog-sig, When we initially implemented file upload to PyPI it was our intention that the file be immutable once uploaded. The goal was to make things significantly simpler for end users - there would only ever be one file with a given name. If the content changed then so must the name (typically by creating a new release version.) After the upload facility was put in place we also added the ability to delete files uploaded to pypi. This created a loophole: if a package owner knew how to they could delete the file and re-upload, thus circumventing the replacement protection. I'm considering closing this loophole by retaining a record of the uploaded file (though not the contents) so that future uploads with the same name wouldn't be allowed. I understand that this is how the ruby gem archive handles deletion of files. Your thoughts? Richard From robertc at robertcollins.net Mon Jan 30 00:59:09 2012 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 30 Jan 2012 12:59:09 +1300 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: Message-ID: On Mon, Jan 30, 2012 at 12:47 PM, Richard Jones wrote: > I'm considering closing this loophole by retaining a record of the > uploaded file (though not the contents) so that future uploads with > the same name wouldn't be allowed. I understand that this is how the > ruby gem archive handles deletion of files. Please allow for never-downloaded files to be replaced; or perhaps some low threshold (like 2 or 3) downloads. Its handy when a bad upload is made to just-fix-it. -Rob From richard at python.org Mon Jan 30 01:05:41 2012 From: richard at python.org (Richard Jones) Date: Mon, 30 Jan 2012 11:05:41 +1100 Subject: [Catalog-sig] Classifier: Python 3.3 In-Reply-To: <4F203243.4080200@jcea.es> References: <4F203243.4080200@jcea.es> Message-ID: On 26 January 2012 03:48, Jesus Cea wrote: > I test my code against current 3.3 head, and I would like to mark > these releases as "3.3 ready", but PYPI doesn't accept it :-). Added. Richard From richard at python.org Mon Jan 30 01:06:50 2012 From: richard at python.org (Richard Jones) Date: Mon, 30 Jan 2012 11:06:50 +1100 Subject: [Catalog-sig] New MPL 2.0 license classifier In-Reply-To: References: Message-ID: On 24 January 2012 22:03, Hanno Schlichting wrote: > Could we get a new classifier for the MPL 2.0 license as per > https://mpl.mozilla.org/2012/01/03/announcing-mpl-2-0 and > http://www.opensource.org/licenses/MPL-2.0. > > I think it should be: > > License :: OSI Approved :: Mozilla Public License 2.0 (MPL) Added. Richard From robertc at robertcollins.net Mon Jan 30 01:27:27 2012 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 30 Jan 2012 13:27:27 +1300 Subject: [Catalog-sig] trove - LGPL v3 not recognised? In-Reply-To: References: <4EC035BC.2040905@v.loewis.de> <20111114003747.GG2861@unaka.lan> <20111115012828.GK2861@unaka.lan> <4EC20ADE.80500@v.loewis.de> <20111115222849.GN2861@unaka.lan> Message-ID: On Mon, Dec 12, 2011 at 6:41 AM, Robert Collins wrote: > These are the ones I would like to see: > >> >License :: OSI Approved :: GNU Lesser General Public License v2 (LGPLv2) >> >License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3) >> >License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+) >> >License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+) >> >License :: OSI Approved :: GNU General Public License v2 (GPLv2) >> >License :: OSI Approved :: GNU General Public License v3 (GPLv3) >> >License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+) >> >License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) > > Doing a 4th level would raise a bunch of interactions between legal > intent ?- the above list seems like a good compromise. We've just had > a user turn up confused around this again, so - do we have sufficient > consensus to add these? > > -Rob Ping? I'm not sure how to check if these were added or not ? -Rob From martin at v.loewis.de Mon Jan 30 01:38:00 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Jan 2012 01:38:00 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: Message-ID: <4F25E668.2040603@v.loewis.de> > When we initially implemented file upload to PyPI it was our intention > that the file be immutable once uploaded. The goal was to make things > significantly simpler for end users - there would only ever be one > file with a given name. If the content changed then so must the name > (typically by creating a new release version.) I don't actually recall that being a goal :-) > Your thoughts? -1. There are plenty of ways to check whether the file was modified if you already have a copy of it. Users just need to accept that files may change, and package authors need to accept that users may retain old copies of a file even after they replaced it. I just got a user comment a week ago of a user explicitly thanking about the ability to replace files after already publishing them. Regards, Martin From richard at python.org Mon Jan 30 01:41:31 2012 From: richard at python.org (Richard Jones) Date: Mon, 30 Jan 2012 11:41:31 +1100 Subject: [Catalog-sig] trove - LGPL v3 not recognised? In-Reply-To: References: <4EC035BC.2040905@v.loewis.de> <20111114003747.GG2861@unaka.lan> <20111115012828.GK2861@unaka.lan> <4EC20ADE.80500@v.loewis.de> <20111115222849.GN2861@unaka.lan> Message-ID: On 30 January 2012 11:27, Robert Collins wrote: > On Mon, Dec 12, 2011 at 6:41 AM, Robert Collins > wrote: >> These are the ones I would like to see: >> >>> >License :: OSI Approved :: GNU Lesser General Public License v2 (LGPLv2) >>> >License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3) >>> >License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+) >>> >License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+) >>> >License :: OSI Approved :: GNU General Public License v2 (GPLv2) >>> >License :: OSI Approved :: GNU General Public License v3 (GPLv3) >>> >License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+) >>> >License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) >> >> Doing a 4th level would raise a bunch of interactions between legal >> intent ?- the above list seems like a good compromise. We've just had >> a user turn up confused around this again, so - do we have sufficient >> consensus to add these? >> >> -Rob > > Ping? I'm not sure how to check if these were added or not ? The classifier list is available on PyPI via the "List trove classifiers" sidebar link: http://pypi.python.org/pypi?%3Aaction=list_classifiers I've not been following the discussion closely so I'm not sure that a consensus has been agreed upon (I'm guessing that the participants in the November discussion went on vacation.) Richard From donald.stufft at gmail.com Mon Jan 30 01:43:02 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 29 Jan 2012 19:43:02 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F25E668.2040603@v.loewis.de> References: <4F25E668.2040603@v.loewis.de> Message-ID: <147E0BF205A242D5A14A5B672C2831E2@gmail.com> I'm very much +1 on this. I don't see any use case for modifying a file after it was already released. It means that if I install version 1.0 of a package today, tomorrow version 1.0 of a package might be different and introduce incompatibilities. It lowers the ability of anyone using PyPI or it's mirrors to be secure in the fact that when they tested their application with versions X,Y,Z of a library, that it should continue to work exactly the same with versions X,Y,Z as a library. There isn't a limited set of version numbers, if someone makes a mistake on packaging the can delete the file, and increase the version number. On Sunday, January 29, 2012 at 7:38 PM, "Martin v. L?wis" wrote: > > When we initially implemented file upload to PyPI it was our intention > > that the file be immutable once uploaded. The goal was to make things > > significantly simpler for end users - there would only ever be one > > file with a given name. If the content changed then so must the name > > (typically by creating a new release version.) > > > > > I don't actually recall that being a goal :-) > > > Your thoughts? > > -1. There are plenty of ways to check whether the file was modified if > you already have a copy of it. Users just need to accept that files may > change, and package authors need to accept that users may retain old > copies of a file even after they replaced it. > > I just got a user comment a week ago of a user explicitly thanking about > the ability to replace files after already publishing them. > > Regards, > Martin > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Mon Jan 30 01:44:22 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 29 Jan 2012 19:44:22 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F25E668.2040603@v.loewis.de> References: <4F25E668.2040603@v.loewis.de> Message-ID: On Sunday, January 29, 2012 at 7:38 PM, "Martin v. L?wis" wrote: > > When we initially implemented file upload to PyPI it was our intention > > that the file be immutable once uploaded. The goal was to make things > > significantly simpler for end users - there would only ever be one > > file with a given name. If the content changed then so must the name > > (typically by creating a new release version.) > > > > > I don't actually recall that being a goal :-) > > > Your thoughts? > > -1. There are plenty of ways to check whether the file was modified if > you already have a copy of it. Users just need to accept that files may > change, and package authors need to accept that users may retain old > copies of a file even after they replaced it. > > I don't always have a copy of the file, I might only have a reference such as slumber==0.3.0. > > I just got a user comment a week ago of a user explicitly thanking about > the ability to replace files after already publishing them. > > Regards, > Martin > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r1chardj0n3s at gmail.com Mon Jan 30 01:46:31 2012 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Mon, 30 Jan 2012 11:46:31 +1100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: Message-ID: On 30 January 2012 10:59, Robert Collins wrote: > On Mon, Jan 30, 2012 at 12:47 PM, Richard Jones wrote: >> I'm considering closing this loophole by retaining a record of the >> uploaded file (though not the contents) so that future uploads with >> the same name wouldn't be allowed. I understand that this is how the >> ruby gem archive handles deletion of files. > > Please allow for never-downloaded files to be replaced; or perhaps > some low threshold (like 2 or 3) downloads. Its handy when a bad > upload is made to just-fix-it. This is tricky: download counts are only tallied once every 24 hours using the local web server logs and grabbing the download count files from the mirrors. Richard From thomas at thomas-lotze.de Mon Jan 30 07:26:46 2012 From: thomas at thomas-lotze.de (Thomas Lotze) Date: Mon, 30 Jan 2012 07:26:46 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole References: Message-ID: Richard Jones wrote: > I'm considering closing this loophole by retaining a record of the > uploaded file (though not the contents) so that future uploads with the > same name wouldn't be allowed. I understand that this is how the ruby gem > archive handles deletion of files. I'd even suggest disallowing to delete files in the first place and retain them including their contents. I regularly see trouble arising from files having been deleted from PyPI that are needed even after their authors considered them obsolete. This may simply be due to version pinning in some application deployment or similar. -- Thomas From r1chardj0n3s at gmail.com Mon Jan 30 08:10:50 2012 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Mon, 30 Jan 2012 18:10:50 +1100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: Message-ID: This has been discussed previously (see the mailing list archive.) As a matter of policy we will always allow users to delete their content from pypi. On Jan 30, 2012 5:26 PM, "Thomas Lotze" wrote: > Richard Jones wrote: > > > I'm considering closing this loophole by retaining a record of the > > uploaded file (though not the contents) so that future uploads with the > > same name wouldn't be allowed. I understand that this is how the ruby gem > > archive handles deletion of files. > > I'd even suggest disallowing to delete files in the first place and > retain them including their contents. I regularly see trouble arising from > files having been deleted from PyPI that are needed even after their > authors considered them obsolete. This may simply be due to version > pinning in some application deployment or similar. > > -- > Thomas > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 30 09:04:05 2012 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 30 Jan 2012 09:04:05 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F25E668.2040603@v.loewis.de> Message-ID: <4F264EF5.1070102@v.loewis.de> >> -1. There are plenty of ways to check whether the file was modified if >> you already have a copy of it. Users just need to accept that files may >> change, and package authors need to accept that users may retain old >> copies of a file even after they replaced it. > I don't always have a copy of the file, I might only have a reference > such as slumber==0.3.0. The better. A responsible author, when replacing an existing file, should make sure that it is reasonably compatible with the previous copy of the file. E.g. the update may include corrected typos or include files that the previous copy didn't include; the previous copy may have actually not worked at all in some circumstances. Now, it may be that the author does break your code by mistake when replacing a file. You should then report that to the author, asking him to restore the original file and be more careful in the future. Regards, Martin From donald.stufft at gmail.com Mon Jan 30 09:43:07 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 30 Jan 2012 03:43:07 -0500 Subject: [Catalog-sig] Fw: Proposal: close the PyPI file-replacement loophole In-Reply-To: <3E935F1A095340458EB7EB5C0BF419AC@gmail.com> References: <4F25E668.2040603@v.loewis.de> <4F264EF5.1070102@v.loewis.de> <3E935F1A095340458EB7EB5C0BF419AC@gmail.com> Message-ID: <08910B7104F046078DBE3F58981AB039@gmail.com> Forwarding as I mistakingly sent this directly to Matin, sorry! Forwarded message: > From: Donald Stufft > To: "Martin v. L?wis" > Date: Monday, January 30, 2012 3:36:09 AM > Subject: Re: [Catalog-sig] Proposal: close the PyPI file-replacement loophole > > A Major goal in any deployment/installation system is reproducible builds. Allowing re uploads directly goes against this goal. I (and a lot of Python developers I would imagine) purposely pin to specific versions because we *know* those versions work. Currently if I rely on PyPI or any of it's mirrors I just have to cross my fingers and hope that the file i'm getting is the file that I tested against. > > A responsible author wouldn't change his files after people are using it. Unfortunately not all authors are responsible which is why restrictions should be put in place. > > I think there are very clear bad things that could happen due to mutable packages, I can't think of a single bad thing that could happen due to immutable packages other than "if the author messed something up he might have to increase his version number". Increasing a version number is a very minor problem compared to breaking software. > > So my questions to you are: > > 1. What is the worst case if packages are made immutable? > 2. What is the worst case if they are kept mutable? > 3. Best case for immutable? > 4. Best case for mutable? > > That I can think of it's: 1) Author Might have to "waste" a version number uploading a fix 2) Author might break (or introduce major security vulnerabilities), inadvertently or otherwise exiting software 3) People depending on packages can use PyPI and be secure in the fact that what they got today will be the same as what they get tomorrow 4) People depending on packages can get "secret" bug fixes. > > Between the two the worst case for immutable is basically a noop, and the worst case for mutable is a very serious problem which leads many people to needlessly abandon PyPI for when installing packages matter and use their own internal systems. I very strongly feel that the worst case for mutable is a serious problem and it outweighs the very minor benefit package authors get from being able to re upload. > > On an additional note, a good compromise might be to allow reuploads for the first 30 minutes or an hour, and after that prevent it. You still provide that minor benefit in the only situation it's a valid use in my opinion (the "oh no I just uploaded a package and it was broken"), but you let people be secure in the fact that when I test my software against a specific version, I can install that version over and over again and get the same results. > > On Monday, January 30, 2012 at 3:04 AM, "Martin v. L?wis" wrote: > > > > -1. There are plenty of ways to check whether the file was modified if > > > > you already have a copy of it. Users just need to accept that files may > > > > change, and package authors need to accept that users may retain old > > > > copies of a file even after they replaced it. > > > > > > > > > > I don't always have a copy of the file, I might only have a reference > > > such as slumber==0.3.0. > > > > > > > > > The better. A responsible author, when replacing an existing file, > > should make sure that it is reasonably compatible with the previous > > copy of the file. E.g. the update may include corrected typos or include > > files that the previous copy didn't include; the previous copy may have > > actually not worked at all in some circumstances. > > > > Now, it may be that the author does break your code by mistake when > > replacing a file. You should then report that to the author, asking > > him to restore the original file and be more careful in the future. > > > > Regards, > > Martin > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Jan 30 10:23:23 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 30 Jan 2012 10:23:23 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: Message-ID: <4F26618B.8030100@egenix.com> Richard Jones wrote: > Hi catalog-sig, > > When we initially implemented file upload to PyPI it was our intention > that the file be immutable once uploaded. The goal was to make things > significantly simpler for end users - there would only ever be one > file with a given name. If the content changed then so must the name > (typically by creating a new release version.) > > After the upload facility was put in place we also added the ability > to delete files uploaded to pypi. This created a loophole: if a > package owner knew how to they could delete the file and re-upload, > thus circumventing the replacement protection. > > I'm considering closing this loophole by retaining a record of the > uploaded file (though not the contents) so that future uploads with > the same name wouldn't be allowed. I understand that this is how the > ruby gem archive handles deletion of files. > > Your thoughts? I don't think that's a good idea, since it would require the package author to issue a new release whenever something goes wrong with an upload (e.g. missing files, corrupted archive, etc.). Please leave the existing logic in place. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Mon Jan 30 10:26:04 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 30 Jan 2012 04:26:04 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26618B.8030100@egenix.com> References: <4F26618B.8030100@egenix.com> Message-ID: <5AFEE304CF20438284D88082D4D07044@gmail.com> On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: > Richard Jones wrote: > > Hi catalog-sig, > > > > When we initially implemented file upload to PyPI it was our intention > > that the file be immutable once uploaded. The goal was to make things > > significantly simpler for end users - there would only ever be one > > file with a given name. If the content changed then so must the name > > (typically by creating a new release version.) > > > > After the upload facility was put in place we also added the ability > > to delete files uploaded to pypi. This created a loophole: if a > > package owner knew how to they could delete the file and re-upload, > > thus circumventing the replacement protection. > > > > I'm considering closing this loophole by retaining a record of the > > uploaded file (though not the contents) so that future uploads with > > the same name wouldn't be allowed. I understand that this is how the > > ruby gem archive handles deletion of files. > > > > Your thoughts? > > I don't think that's a good idea, since it would require the > package author to issue a new release whenever something goes wrong > with an upload (e.g. missing files, corrupted archive, etc.). > > Please leave the existing logic in place. And version numbers are a scarce resource? (Even though I believe it would be acceptable to cover that particular use case by giving a grace period of when you can re upload). > > Thanks, > -- > Marc-Andre Lemburg > eGenix.com (http://eGenix.com) > > Professional Python Services directly from the Source (#1, Jan 30 2012) > > > > Python/Zope Consulting and Support ... http://www.egenix.com/ > > > > mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drkjam at gmail.com Mon Jan 30 10:14:12 2012 From: drkjam at gmail.com (drkjam) Date: Mon, 30 Jan 2012 09:14:12 +0000 Subject: [Catalog-sig] Fw: Proposal: close the PyPI file-replacement loophole In-Reply-To: <08910B7104F046078DBE3F58981AB039@gmail.com> References: <4F25E668.2040603@v.loewis.de> <4F264EF5.1070102@v.loewis.de> <3E935F1A095340458EB7EB5C0BF419AC@gmail.com> <08910B7104F046078DBE3F58981AB039@gmail.com> Message-ID: <67C92840-B496-4869-904F-EA5C147CD9C0@gmail.com> +1 on immutability once a distribution is uploaded to PyPI. The benefits far outweigh the drawbacks. Catering for the "overwrite within a certain time window" scenario just serves to complicate what should be a very simple rule. Even though PyPI acts as a passthrough gateway to other file sources e.g. github this shouldn't deter us from aspiring to provide users with greater confidence in the files that are hosted directly on PyPI by making this change. On 30 Jan 2012, at 08:43, Donald Stufft wrote: > Forwarding as I mistakingly sent this directly to Matin, sorry! > Forwarded message: > >> From: Donald Stufft >> To: "Martin v. L?wis" >> Date: Monday, January 30, 2012 3:36:09 AM >> Subject: Re: [Catalog-sig] Proposal: close the PyPI file-replacement loophole >> >> A Major goal in any deployment/installation system is reproducible builds. Allowing re uploads directly goes against this goal. I (and a lot of Python developers I would imagine) purposely pin to specific versions because we *know* those versions work. Currently if I rely on PyPI or any of it's mirrors I just have to cross my fingers and hope that the file i'm getting is the file that I tested against. >> >> A responsible author wouldn't change his files after people are using it. Unfortunately not all authors are responsible which is why restrictions should be put in place. >> >> I think there are very clear bad things that could happen due to mutable packages, I can't think of a single bad thing that could happen due to immutable packages other than "if the author messed something up he might have to increase his version number". Increasing a version number is a very minor problem compared to breaking software. >> >> So my questions to you are: >> >> 1. What is the worst case if packages are made immutable? >> 2. What is the worst case if they are kept mutable? >> 3. Best case for immutable? >> 4. Best case for mutable? >> >> That I can think of it's: 1) Author Might have to "waste" a version number uploading a fix 2) Author might break (or introduce major security vulnerabilities), inadvertently or otherwise exiting software 3) People depending on packages can use PyPI and be secure in the fact that what they got today will be the same as what they get tomorrow 4) People depending on packages can get "secret" bug fixes. >> >> Between the two the worst case for immutable is basically a noop, and the worst case for mutable is a very serious problem which leads many people to needlessly abandon PyPI for when installing packages matter and use their own internal systems. I very strongly feel that the worst case for mutable is a serious problem and it outweighs the very minor benefit package authors get from being able to re upload. >> >> On an additional note, a good compromise might be to allow reuploads for the first 30 minutes or an hour, and after that prevent it. You still provide that minor benefit in the only situation it's a valid use in my opinion (the "oh no I just uploaded a package and it was broken"), but you let people be secure in the fact that when I test my software against a specific version, I can install that version over and over again and get the same results. >> >> On Monday, January 30, 2012 at 3:04 AM, "Martin v. L?wis" wrote: >>>>> -1. There are plenty of ways to check whether the file was modified if >>>>> you already have a copy of it. Users just need to accept that files may >>>>> change, and package authors need to accept that users may retain old >>>>> copies of a file even after they replaced it. >>>> I don't always have a copy of the file, I might only have a reference >>>> such as slumber==0.3.0. >>> >>> The better. A responsible author, when replacing an existing file, >>> should make sure that it is reasonably compatible with the previous >>> copy of the file. E.g. the update may include corrected typos or include >>> files that the previous copy didn't include; the previous copy may have >>> actually not worked at all in some circumstances. >>> >>> Now, it may be that the author does break your code by mistake when >>> replacing a file. You should then report that to the author, asking >>> him to restore the original file and be more careful in the future. >>> >>> Regards, >>> Martin >> > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Jan 30 10:46:00 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 30 Jan 2012 10:46:00 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <5AFEE304CF20438284D88082D4D07044@gmail.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> Message-ID: <4F2666D8.60905@egenix.com> Donald Stufft wrote: > > > On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: > >> Richard Jones wrote: >>> Hi catalog-sig, >>> >>> When we initially implemented file upload to PyPI it was our intention >>> that the file be immutable once uploaded. The goal was to make things >>> significantly simpler for end users - there would only ever be one >>> file with a given name. If the content changed then so must the name >>> (typically by creating a new release version.) >>> >>> After the upload facility was put in place we also added the ability >>> to delete files uploaded to pypi. This created a loophole: if a >>> package owner knew how to they could delete the file and re-upload, >>> thus circumventing the replacement protection. >>> >>> I'm considering closing this loophole by retaining a record of the >>> uploaded file (though not the contents) so that future uploads with >>> the same name wouldn't be allowed. I understand that this is how the >>> ruby gem archive handles deletion of files. >>> >>> Your thoughts? >> >> I don't think that's a good idea, since it would require the >> package author to issue a new release whenever something goes wrong >> with an upload (e.g. missing files, corrupted archive, etc.). >> >> Please leave the existing logic in place. > And version numbers are a scarce resource? No, but having to kick off the whole release process again just because something went wrong when uploading release files to PyPI causes plenty of trouble. > (Even though I believe it would be acceptable to cover that particular use case by giving a grace period of when you can re upload). Can't we just leave dealing with that problem to the package authors ? It's their responsibility, not PyPI's. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Mon Jan 30 11:06:19 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 30 Jan 2012 05:06:19 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F2666D8.60905@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> Message-ID: <91D1391E392F43858A8047E3EC5D6BBE@gmail.com> On Monday, January 30, 2012 at 4:46 AM, M.-A. Lemburg wrote: > Donald Stufft wrote: > > > > > > On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: > > > > > Richard Jones wrote: > > > > Hi catalog-sig, > > > > > > > > When we initially implemented file upload to PyPI it was our intention > > > > that the file be immutable once uploaded. The goal was to make things > > > > significantly simpler for end users - there would only ever be one > > > > file with a given name. If the content changed then so must the name > > > > (typically by creating a new release version.) > > > > > > > > After the upload facility was put in place we also added the ability > > > > to delete files uploaded to pypi. This created a loophole: if a > > > > package owner knew how to they could delete the file and re-upload, > > > > thus circumventing the replacement protection. > > > > > > > > I'm considering closing this loophole by retaining a record of the > > > > uploaded file (though not the contents) so that future uploads with > > > > the same name wouldn't be allowed. I understand that this is how the > > > > ruby gem archive handles deletion of files. > > > > > > > > Your thoughts? > > > > > > I don't think that's a good idea, since it would require the > > > package author to issue a new release whenever something goes wrong > > > with an upload (e.g. missing files, corrupted archive, etc.). > > > > > > Please leave the existing logic in place. > > And version numbers are a scarce resource? > > > > > No, but having to kick off the whole release process again > just because something went wrong when uploading release files > to PyPI causes plenty of trouble. > I would assert that almost every time something goes wrong with "uploading to PyPI" it's actually "I didn't package my software correctly". A better solution to people failing package correctly (missing MANIFEST, whatever) is to test your package prior to uploading to PyPI. Then you don't need mutable files, your release process becomes more robust, your releases become more robust, and the ecosystem in general becomes more robust. Further more I would still argue that the benefits to the community outweigh the ability for people to skimp on the release process. Either you are doing your releases adhoc, in that case you don't have much of a release process to begin with, so doing it over again to bump it up one more isn't a huge deal, or you have a large release process and testing the package before distributing it should be a part of it. > > > (Even though I believe it would be acceptable to cover that particular use case by giving a grace period of when you can re upload). > > > Can't we just leave dealing with that problem to the package authors ? > It's their responsibility, not PyPI's. > > In my opinion No. PyPI is acting as the central repository, it is it's responsibility to take a reasonable effort to protect the people that depend on it. The current solution doesn't just make end developers at risk from a bad author breaking their well tested software. It also puts the security of their software under the the author's watch. Author of a particular package's credentials get leaked/stolen/whatever? suddenly my software is now possibly vulnerable to whatever person did it decides to upload. It puts the integrity of my (proverbial my) software in the hands of a disparate group of authors who may or may not have the same stringent testing that I do. Any python application that get's installed from PyPI is at risk of mysteriously breaking, even with a "known good" configuration. These bugs are often hard to track down, and very confusing and difficult to determine why they are occurring when they never did before. > > -- > Marc-Andre Lemburg > eGenix.com (http://eGenix.com) > > Professional Python Services directly from the Source (#1, Jan 30 2012) > > > > Python/Zope Consulting and Support ... http://www.egenix.com/ > > > > mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Jan 30 11:27:11 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 30 Jan 2012 11:27:11 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <91D1391E392F43858A8047E3EC5D6BBE@gmail.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <91D1391E392F43858A8047E3EC5D6BBE@gmail.com> Message-ID: <4F26707F.3020909@egenix.com> Donald Stufft wrote: > On Monday, January 30, 2012 at 4:46 AM, M.-A. Lemburg wrote: >> Donald Stufft wrote: >>> >>> >>> On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: >>> >>>> Richard Jones wrote: >>>>> Hi catalog-sig, >>>>> >>>>> When we initially implemented file upload to PyPI it was our intention >>>>> that the file be immutable once uploaded. The goal was to make things >>>>> significantly simpler for end users - there would only ever be one >>>>> file with a given name. If the content changed then so must the name >>>>> (typically by creating a new release version.) >>>>> >>>>> After the upload facility was put in place we also added the ability >>>>> to delete files uploaded to pypi. This created a loophole: if a >>>>> package owner knew how to they could delete the file and re-upload, >>>>> thus circumventing the replacement protection. >>>>> >>>>> I'm considering closing this loophole by retaining a record of the >>>>> uploaded file (though not the contents) so that future uploads with >>>>> the same name wouldn't be allowed. I understand that this is how the >>>>> ruby gem archive handles deletion of files. >>>>> >>>>> Your thoughts? >>>> >>>> I don't think that's a good idea, since it would require the >>>> package author to issue a new release whenever something goes wrong >>>> with an upload (e.g. missing files, corrupted archive, etc.). >>>> >>>> Please leave the existing logic in place. >>> And version numbers are a scarce resource? >>> >> >> >> No, but having to kick off the whole release process again >> just because something went wrong when uploading release files >> to PyPI causes plenty of trouble. >> > I would assert that almost every time something goes wrong with "uploading to PyPI" it's actually "I didn't package my software correctly". A better solution to people failing package correctly (missing MANIFEST, whatever) is to test your package prior to uploading to PyPI. Then you don't need mutable files, your release process becomes more robust, your releases become more robust, and the ecosystem in general becomes more robust. > > Further more I would still argue that the benefits to the community outweigh the ability for people to skimp on the release process. Either you are doing your releases adhoc, in that case you don't have much of a release process to begin with, so doing it over again to bump it up one more isn't a huge deal, or you have a large release process and testing the package before distributing it should be a part of it. Due to the way PyPI uploads through distutils work by default, it is not always easy to apply those checks to the uploaded files (distutils recreates the distribution files when running the upload command and even though it is possible to have the command reuse an already created distribution file, that process is tricky and not well known). Besides, we're not talking about a common case here, just an emergency exit that can be used if needed. >>> (Even though I believe it would be acceptable to cover that particular use case by giving a grace period of when you can re upload). >> >> >> Can't we just leave dealing with that problem to the package authors ? >> It's their responsibility, not PyPI's. >> >> > > In my opinion No. PyPI is acting as the central repository, it is it's responsibility to take a reasonable effort to protect the people that depend on it. The current solution doesn't just make end developers at risk from a bad author breaking their well tested software. It also puts the security of their software under the the author's watch. Author of a particular package's credentials get leaked/stolen/whatever? suddenly my software is now possibly vulnerable to whatever person did it decides to upload. > > It puts the integrity of my (proverbial my) software in the hands of a disparate group of authors who may or may not have the same stringent testing that I do. Any python application that get's installed from PyPI is at risk of mysteriously breaking, even with a "known good" configuration. These bugs are often hard to track down, and very confusing and difficult to determine why they are occurring when they never did before. PyPI uploads get stored with a hash sum, so any such changes can easily be recognized on the client side, if there's a need. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ubershmekel at gmail.com Mon Jan 30 11:46:26 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 30 Jan 2012 12:46:26 +0200 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26707F.3020909@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <91D1391E392F43858A8047E3EC5D6BBE@gmail.com> <4F26707F.3020909@egenix.com> Message-ID: On Mon, Jan 30, 2012 at 12:27 PM, M.-A. Lemburg wrote: > Besides, we're not talking about a common case here, just an emergency > exit that can be used if needed. > > This rare "emergency" can be handled by emailing a pypi admin. It most certainly isn't worth the very real and global security and reliability risks. Most cases won't email a pypi admin as it's just that easy to increment the version by an 0.0.1 and the fact that it probably isn't an emergency to begin with. Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Mon Jan 30 12:07:10 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 30 Jan 2012 06:07:10 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: Message-ID: On Sun, Jan 29, 2012 at 6:47 PM, Richard Jones wrote: > Hi catalog-sig, > > When we initially implemented file upload to PyPI it was our intention > that the file be immutable once uploaded. The goal was to make things > significantly simpler for end users - there would only ever be one > file with a given name. If the content changed then so must the name > (typically by creating a new release version.) > > After the upload facility was put in place we also added the ability > to delete files uploaded to pypi. This created a loophole: if a > package owner knew how to they could delete the file and re-upload, > thus circumventing the replacement protection. > > I'm considering closing this loophole by retaining a record of the > uploaded file (though not the contents) so that future uploads with > the same name wouldn't be allowed. I understand that this is how the > ruby gem archive handles deletion of files. +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From ziade.tarek at gmail.com Mon Jan 30 19:08:22 2012 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 30 Jan 2012 10:08:22 -0800 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F2666D8.60905@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> Message-ID: On Mon, Jan 30, 2012 at 1:46 AM, M.-A. Lemburg wrote: > Donald Stufft wrote: > > > > > > On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: > > > >> Richard Jones wrote: > >>> Hi catalog-sig, > >>> > >>> When we initially implemented file upload to PyPI it was our intention > >>> that the file be immutable once uploaded. The goal was to make things > >>> significantly simpler for end users - there would only ever be one > >>> file with a given name. If the content changed then so must the name > >>> (typically by creating a new release version.) > >>> > >>> After the upload facility was put in place we also added the ability > >>> to delete files uploaded to pypi. This created a loophole: if a > >>> package owner knew how to they could delete the file and re-upload, > >>> thus circumventing the replacement protection. > >>> > >>> I'm considering closing this loophole by retaining a record of the > >>> uploaded file (though not the contents) so that future uploads with > >>> the same name wouldn't be allowed. I understand that this is how the > >>> ruby gem archive handles deletion of files. > >>> > >>> Your thoughts? > >> > >> I don't think that's a good idea, since it would require the > >> package author to issue a new release whenever something goes wrong > >> with an upload (e.g. missing files, corrupted archive, etc.). > >> > >> Please leave the existing logic in place. > > And version numbers are a scarce resource? > > No, but having to kick off the whole release process again > just because something went wrong when uploading release files > to PyPI causes plenty of trouble. > It's the opposite that gets you into trouble: once you have uploaded something at PyPI, it potentially gets copied to mirrors. The mirror protocol, as far as I remember, does not deal with 'updates of existing files' IOW if the release is broken and you fix it in pypi, it might stay broken in mirrorsand the inconsistent state is much more trouble. so +1 for creating a new release whatever state the previous published one is in - release numbers are not expensive what about adding a metadata flag to releases ? e.g. "deprecated" - that way client tools know they need to avoid this one and developers can change the flag > > (Even though I believe it would be acceptable to cover that particular > use case by giving a grace period of when you can re upload). > > Can't we just leave dealing with that problem to the package authors ? > It's their responsibility, not PyPI's. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 30 2012) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Tarek Ziad? | http://ziade.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Jan 30 19:27:28 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 30 Jan 2012 19:27:28 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> Message-ID: <4F26E110.9090307@egenix.com> Tarek Ziad? wrote: > On Mon, Jan 30, 2012 at 1:46 AM, M.-A. Lemburg wrote: > >> Donald Stufft wrote: >>> >>> >>> On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: >>> >>>> Richard Jones wrote: >>>>> Hi catalog-sig, >>>>> >>>>> When we initially implemented file upload to PyPI it was our intention >>>>> that the file be immutable once uploaded. The goal was to make things >>>>> significantly simpler for end users - there would only ever be one >>>>> file with a given name. If the content changed then so must the name >>>>> (typically by creating a new release version.) >>>>> >>>>> After the upload facility was put in place we also added the ability >>>>> to delete files uploaded to pypi. This created a loophole: if a >>>>> package owner knew how to they could delete the file and re-upload, >>>>> thus circumventing the replacement protection. >>>>> >>>>> I'm considering closing this loophole by retaining a record of the >>>>> uploaded file (though not the contents) so that future uploads with >>>>> the same name wouldn't be allowed. I understand that this is how the >>>>> ruby gem archive handles deletion of files. >>>>> >>>>> Your thoughts? >>>> >>>> I don't think that's a good idea, since it would require the >>>> package author to issue a new release whenever something goes wrong >>>> with an upload (e.g. missing files, corrupted archive, etc.). >>>> >>>> Please leave the existing logic in place. >>> And version numbers are a scarce resource? >> >> No, but having to kick off the whole release process again >> just because something went wrong when uploading release files >> to PyPI causes plenty of trouble. >> > > It's the opposite that gets you into trouble: once you have uploaded > something at PyPI, it potentially gets copied to mirrors. > > The mirror protocol, as far as I remember, does not deal with 'updates of > existing files' > > IOW if the release is broken and you fix it in pypi, it might stay broken > in mirrorsand the inconsistent state is much more trouble. That shouldn't be a problem if the mirrors correctly implements the protocol: PEP 381: """ Mirrors must reduce the amount of data transfered between the central server and the mirror. To achieve that, they MUST use the changelog() PyPI XML-RPC call, and only refetch the packages that have been changed since the last time. For each package P, they MUST copy documents /simple/P/ and /serversig/P. If a package is deleted on the central server, they MUST delete the package and all associated files. To detect modification of package files, they MAY cache the file's ETag, and MAY request skipping it using the If-none-match header. """ Note that the whole package (including all files) is refetched whenever something changes. The only allowed optimization is to look at the file's modification date, which will be different for newly uploaded files of the same name. See Martin's pep381client for details: https://bitbucket.org/loewis/pep381client/src/9d55326ed555/pep381client/__init__.py -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Mon Jan 30 20:08:22 2012 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 30 Jan 2012 11:08:22 -0800 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26E110.9090307@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> Message-ID: ok, I did not remember that :) Nevertheless, you still have the case where someone gets the "old version" of Foo 1.2, in his environment, and won't get the new version of Foo 1.2, so it won't work for her and she will not understand why I think it's a very bad idea to let a window where two different versions of the same ...version are in the wild, even if it's a very small window of time. This happened to me in the past : I pushed a version, screwed up, delete it to push the same version within minutes, and made some people mad because they were stocked with the old non-working version :) while this is the developer responsibility not to screw things like this, we should make it not possible by design. The caveat of doing a new release is minimal if people do the proper automation work of their release process. On Mon, Jan 30, 2012 at 10:27 AM, M.-A. Lemburg wrote: > Tarek Ziad? wrote: > > On Mon, Jan 30, 2012 at 1:46 AM, M.-A. Lemburg wrote: > > > >> Donald Stufft wrote: > >>> > >>> > >>> On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: > >>> > >>>> Richard Jones wrote: > >>>>> Hi catalog-sig, > >>>>> > >>>>> When we initially implemented file upload to PyPI it was our > intention > >>>>> that the file be immutable once uploaded. The goal was to make things > >>>>> significantly simpler for end users - there would only ever be one > >>>>> file with a given name. If the content changed then so must the name > >>>>> (typically by creating a new release version.) > >>>>> > >>>>> After the upload facility was put in place we also added the ability > >>>>> to delete files uploaded to pypi. This created a loophole: if a > >>>>> package owner knew how to they could delete the file and re-upload, > >>>>> thus circumventing the replacement protection. > >>>>> > >>>>> I'm considering closing this loophole by retaining a record of the > >>>>> uploaded file (though not the contents) so that future uploads with > >>>>> the same name wouldn't be allowed. I understand that this is how the > >>>>> ruby gem archive handles deletion of files. > >>>>> > >>>>> Your thoughts? > >>>> > >>>> I don't think that's a good idea, since it would require the > >>>> package author to issue a new release whenever something goes wrong > >>>> with an upload (e.g. missing files, corrupted archive, etc.). > >>>> > >>>> Please leave the existing logic in place. > >>> And version numbers are a scarce resource? > >> > >> No, but having to kick off the whole release process again > >> just because something went wrong when uploading release files > >> to PyPI causes plenty of trouble. > >> > > > > It's the opposite that gets you into trouble: once you have uploaded > > something at PyPI, it potentially gets copied to mirrors. > > > > The mirror protocol, as far as I remember, does not deal with 'updates of > > existing files' > > > > IOW if the release is broken and you fix it in pypi, it might stay broken > > in mirrorsand the inconsistent state is much more trouble. > > That shouldn't be a problem if the mirrors correctly implements > the protocol: > > PEP 381: > """ > Mirrors must reduce the amount of data transfered between the central > server and the mirror. To > achieve that, they MUST use the changelog() PyPI XML-RPC call, and only > refetch the packages that > have been changed since the last time. For each package P, they MUST copy > documents /simple/P/ and > /serversig/P. If a package is deleted on the central server, they MUST > delete the package and all > associated files. To detect modification of package files, they MAY cache > the file's ETag, and MAY > request skipping it using the If-none-match header. > """ > > Note that the whole package (including all files) is refetched > whenever something changes. The only allowed optimization is > to look at the file's modification date, which will be different > for newly uploaded files of the same name. > > See Martin's pep381client for details: > > > https://bitbucket.org/loewis/pep381client/src/9d55326ed555/pep381client/__init__.py > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 30 2012) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -- Tarek Ziad? | http://ziade.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Jan 30 20:56:52 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 30 Jan 2012 20:56:52 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> Message-ID: <4F26F604.1040907@egenix.com> Tarek Ziad? wrote: > ok, I did not remember that :) > > Nevertheless, you still have the case where someone gets the "old version" > of Foo 1.2, in his environment, and won't get the new version of Foo 1.2, > so it won't work for her and she will not understand why > > I think it's a very bad idea to let a window where two different versions > of the same ...version are in the wild, even if it's a very small window of > time. > > This happened to me in the past : I pushed a version, screwed up, delete it > to push the same version within minutes, and made some people mad because > they were stocked with the old non-working version :) > > while this is the developer responsibility not to screw things like this, > we should make it not possible by design. The caveat of doing a new release > is minimal if people do the proper automation work of their release process. Sure, I'm not saying that fixes should be done by using the delete/reupload dance. It's always better if you do a dot-release to address the problem. However, there are cases, where you simply don't want a brown bag release to persist and also don't want to issue a new release just to address a problem with e.g. a broken .so or .pyd file in one of the distribution files, rendering it completely broken. Please consider that people are doing releases which require far more than just running "setup.py register upload": releases that require sending out announcements, registering the package at various sites, writing press releases, updating your web site, etc. etc. For those types of packages, doing a quick dot-release is not necessarily an option and replacing a broken distribution file is much less troublesome for both authors and users. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Jan 30 21:07:06 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 30 Jan 2012 21:07:06 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26F604.1040907@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> <4F26F604.1040907@egenix.com> Message-ID: <4F26F86A.6000104@egenix.com> A little off-topic, but I always find it strange that some users of PyPI appear to trust package authors with the software they put up on PyPI, but don't trust them when it comes to the release process. Very strange indeed... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2012) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Mon Jan 30 22:01:51 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 30 Jan 2012 16:01:51 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26F86A.6000104@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> <4F26F604.1040907@egenix.com> <4F26F86A.6000104@egenix.com> Message-ID: <6AF187C1B5CA47C688FEC03F933AC5B0@gmail.com> Writing good software and having good release process don't always go hand in hand. Additionally prior to using a bit of software I can review it and test it. Not the case when the author can replace the file(s) that I reviewed at any time, for any reason he pleases. On Monday, January 30, 2012 at 3:07 PM, M.-A. Lemburg wrote: > A little off-topic, but I always find it strange that some users of PyPI > appear to trust package authors with the software they put up on PyPI, > but don't trust them when it comes to the release process. > Very strange indeed... > > -- > Marc-Andre Lemburg > eGenix.com (http://eGenix.com) > > Professional Python Services directly from the Source (#1, Jan 30 2012) > > > > Python/Zope Consulting and Support ... http://www.egenix.com/ > > > > mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Mon Jan 30 22:14:45 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 30 Jan 2012 21:14:45 +0000 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <6AF187C1B5CA47C688FEC03F933AC5B0@gmail.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> <4F26F604.1040907@egenix.com> <4F26F86A.6000104@egenix.com> <6AF187C1B5CA47C688FEC03F933AC5B0@gmail.com> Message-ID: <4F270845.7060008@simplistix.co.uk> I'm fairly certain PyPI provides MD5 keys for the paranoid... Chris On 30/01/2012 21:01, Donald Stufft wrote: > Writing good software and having good release process don't always go > hand in hand. Additionally prior to using a bit of software I can review > it and test it. Not the case when the author can replace the file(s) > that I reviewed at any time, for any reason he pleases. > > On Monday, January 30, 2012 at 3:07 PM, M.-A. Lemburg wrote: > >> A little off-topic, but I always find it strange that some users of PyPI >> appear to trust package authors with the software they put up on PyPI, >> but don't trust them when it comes to the release process. >> Very strange indeed... >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Jan 30 2012) >>>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: >> >> >> eGenix.com Software, Skills and Services GmbH >> Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________ > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ubershmekel at gmail.com Mon Jan 30 22:27:11 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 30 Jan 2012 23:27:11 +0200 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26F86A.6000104@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> <4F26F604.1040907@egenix.com> <4F26F86A.6000104@egenix.com> Message-ID: On Mon, Jan 30, 2012 at 10:07 PM, M.-A. Lemburg wrote: > A little off-topic, but I always find it strange that some users of PyPI > appear to trust package authors with the software they put up on PyPI, > but don't trust them when it comes to the release process. > Very strange indeed... > > I don't trust "package authors". I do trust specific versions of specific packages that I've tested. If I can't trust PyPI to always give me the exact same result for a specific package-version then I can't use it. IOW if a hacked maintainer account can modify existing releases - PyPI is a very real attack vector into many existing systems. Nothing strange at all, Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 30 22:31:26 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Jan 2012 22:31:26 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> Message-ID: <4F270C2E.8040705@v.loewis.de> > The mirror protocol, as far as I remember, does not deal with 'updates > of existing files' It most certainly does, and pep381client deals with it correctly. > IOW if the release is broken and you fix it in pypi, it might stay > broken in mirrorsand the inconsistent state is much more trouble. No, it won't. The mirrors will notice that the package changed, and recheck all files. The current implementation uses the ETag header to determine whether a file changed, although looking for a changed md5sum might be a better approach. > so +1 for creating a new release whatever state the previous published > one is in - release numbers are not expensive As a guideline to package authors, I can certainly agree with that. I don't mind putting a note in the PyPI UI telling authors that file deletion is consider a violation of the social contract. The issue at hand is whether to disallow recreation of deleted files entirely. I see this as a maintenance nightmare: people first delete the files (which we can't stop them from doing), then try to recreate the file, and find out that this is rejected. Next, they try to delete the release entirely. Not sure whether Richard intended to have the file deletion markage survive that also. If so, they might try to delete the entire package, and re-register that. > what about adding a metadata flag to releases ? e.g. "deprecated" - > that way client tools know they need to avoid this one > and developers can change the flag You mean, if a file is recreated, it is somehow flagged? Sounds fine to me. Regards, Martin From martin at v.loewis.de Mon Jan 30 22:36:32 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Jan 2012 22:36:32 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26F86A.6000104@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> <4F26F604.1040907@egenix.com> <4F26F86A.6000104@egenix.com> Message-ID: <4F270D60.4030303@v.loewis.de> Am 30.01.2012 21:07, schrieb M.-A. Lemburg: > A little off-topic, but I always find it strange that some users of PyPI > appear to trust package authors with the software they put up on PyPI, > but don't trust them when it comes to the release process. > Very strange indeed... My feelings entirely. I'm also shocked at how many people readily use a software whose version number is 0.3.0, and then get upset when they find that 0.4.0 breaks the ABI, or that the author deletes 0.3.0 after 1.0 has been released. Regards, Martin From tjreedy at udel.edu Mon Jan 30 22:37:33 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 30 Jan 2012 16:37:33 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F26707F.3020909@egenix.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <91D1391E392F43858A8047E3EC5D6BBE@gmail.com> <4F26707F.3020909@egenix.com> Message-ID: On 1/30/2012 5:27 AM, M.-A. Lemburg wrote: > Donald Stufft wrote: >> It puts the integrity of my (proverbial my) software in the hands >> of a disparate group of authors who may or may not have the same >> stringent testing that I do. Any python application that get's >> installed from PyPI is at risk of mysteriously breaking, even with >> a "known good" configuration. These bugs are often hard to track >> down, and very confusing and difficult to determine why they are >> occurring when they never did before. > > PyPI uploads get stored with a hash sum, so any such changes can > easily be recognized on the client side, if there's a need. Or redistribute the exact files themselves, as some apps do with cpython. -- Terry Jan Reedy From donald.stufft at gmail.com Mon Jan 30 22:38:31 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 30 Jan 2012 16:38:31 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F270C2E.8040705@v.loewis.de> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F270C2E.8040705@v.loewis.de> Message-ID: On Monday, January 30, 2012 at 4:31 PM, "Martin v. L?wis" wrote: > > The mirror protocol, as far as I remember, does not deal with 'updates > > of existing files' > > > > > It most certainly does, and pep381client deals with it correctly. One of the mirrors doesn't deal with it correctly. I think the one on GAE? Has caused a lot of problems for me. > > > IOW if the release is broken and you fix it in pypi, it might stay > > broken in mirrorsand the inconsistent state is much more trouble. > > > > > No, it won't. The mirrors will notice that the package changed, and > recheck all files. The current implementation uses the ETag header > to determine whether a file changed, although looking for a changed > md5sum might be a better approach. > > > so +1 for creating a new release whatever state the previous published > > one is in - release numbers are not expensive > > > > > As a guideline to package authors, I can certainly agree with that. I > don't mind putting a note in the PyPI UI telling authors that file > deletion is consider a violation of the social contract. > > The issue at hand is whether to disallow recreation of deleted files > entirely. I see this as a maintenance nightmare: people first delete > the files (which we can't stop them from doing), then try to recreate > the file, and find out that this is rejected. > > Offhand I believe deleting the file requires using the web interface? If so it should be trivial to present a warning that states. "Are You Sure You Want To Do This? Note: You Will Not Be Able to Upload This Same FIle Again, You will Need to make a new release". > > Next, they try to delete the release entirely. Not sure whether Richard > intended to have the file deletion markage survive that also. If so, > they might try to delete the entire package, and re-register that. > > It should survive the lifetime of the package. I'd also argue that it should survive past it as this is a different but similar problem. (Someone pulls a _why/mark pilgrim and deletes all their releases, someone else takes this chance to grab the package name, and uploads malicious code. > > > what about adding a metadata flag to releases ? e.g. "deprecated" - > > that way client tools know they need to avoid this one > > and developers can change the flag > > > > > You mean, if a file is recreated, it is somehow flagged? Sounds fine > to me. > > Regards, > Martin > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jan 30 22:43:35 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Jan 2012 22:43:35 +0100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F270845.7060008@simplistix.co.uk> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> <4F26F604.1040907@egenix.com> <4F26F86A.6000104@egenix.com> <6AF187C1B5CA47C688FEC03F933AC5B0@gmail.com> <4F270845.7060008@simplistix.co.uk> Message-ID: <4F270F07.6080803@v.loewis.de> Am 30.01.2012 22:14, schrieb Chris Withers: > I'm fairly certain PyPI provides MD5 keys for the paranoid... Indeed. Users wishing to make sure that the source code they manually reviewed stays the same should really record the md5 of the file, and verify that it is still the same file when downloading it again. It appears that buildout has mechanisms to hard-code the md5sum into the recipe. It would be desirable if other automatic download tools offered similar mechanisms. Regards, Martin From tjreedy at udel.edu Mon Jan 30 22:44:06 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 30 Jan 2012 16:44:06 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <4F264EF5.1070102@v.loewis.de> References: <4F25E668.2040603@v.loewis.de> <4F264EF5.1070102@v.loewis.de> Message-ID: On 1/30/2012 3:04 AM, "Martin v. L?wis" wrote: >>> -1. There are plenty of ways to check whether the file was modified if >>> you already have a copy of it. Users just need to accept that files may >>> change, and package authors need to accept that users may retain old >>> copies of a file even after they replaced it. >> I don't always have a copy of the file, I might only have a reference >> such as slumber==0.3.0. > > The better. A responsible author, when replacing an existing file, > should make sure that it is reasonably compatible with the previous > copy of the file. E.g. the update may include corrected typos or include > files that the previous copy didn't include; the previous copy may have > actually not worked at all in some circumstances. > > Now, it may be that the author does break your code by mistake when > replacing a file. You should then report that to the author, asking > him to restore the original file and be more careful in the future. In book publishing, typographical errors may be corrected in subsequent printing without notice. We do the same thing nightly with the docs. -- Terry Jan Reedy From chris at simplistix.co.uk Mon Jan 30 22:46:25 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 30 Jan 2012 21:46:25 +0000 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> <4F2666D8.60905@egenix.com> <4F26E110.9090307@egenix.com> <4F26F604.1040907@egenix.com> <4F26F86A.6000104@egenix.com> Message-ID: <4F270FB1.4040800@simplistix.co.uk> On 30/01/2012 21:27, Yuval Greenfield wrote: > A little off-topic, but I always find it strange that some users of PyPI > appear to trust package authors with the software they put up on PyPI, > but don't trust them when it comes to the release process. > Very strange indeed... > > > I don't trust "package authors". > > I do trust specific versions of specific packages that I've tested. > > If I can't trust PyPI to always give me the exact same result for a > specific package-version then I can't use it. > > IOW if a hacked maintainer account can modify existing releases - PyPI > is a very real attack vector into many existing systems. Tin foil hats all round ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From richard at python.org Tue Jan 31 01:13:07 2012 From: richard at python.org (Richard Jones) Date: Tue, 31 Jan 2012 11:13:07 +1100 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F25E668.2040603@v.loewis.de> <4F264EF5.1070102@v.loewis.de> Message-ID: On 31 January 2012 08:44, Terry Reedy wrote: > In book publishing, typographical errors may be corrected in subsequent > printing without notice. We do the same thing nightly with the docs. Doc uploads would be unaffected. Richard From pje at telecommunity.com Tue Jan 31 02:37:21 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 30 Jan 2012 20:37:21 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: <5AFEE304CF20438284D88082D4D07044@gmail.com> References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> Message-ID: On Mon, Jan 30, 2012 at 4:26 AM, Donald Stufft wrote: > On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: > > Please leave the existing logic in place. > > And version numbers are a scarce resource? > No, release managers are. ;-) Or more precisely, release management *time* is a scarce resource, and changing the version number in setup.py is far from the only thing you have to do to release a new version of a package. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Jan 31 06:26:09 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 31 Jan 2012 00:26:09 -0500 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> Message-ID: The argument that bumping the release number is hard is an argument that your process should include testing the package before releasing it, not that it should open up anyone using PyPI to install packages for potential damaging security issues on top of the possible random and undefined breakage that can happen for seemingly no reason. On Monday, January 30, 2012 at 8:37 PM, PJ Eby wrote: > On Mon, Jan 30, 2012 at 4:26 AM, Donald Stufft wrote: > > On Monday, January 30, 2012 at 4:23 AM, M.-A. Lemburg wrote: > > > Please leave the existing logic in place. > > > > > > > > > > > > > > > And version numbers are a scarce resource? > > > > > No, release managers are. ;-) > > Or more precisely, release management *time* is a scarce resource, and changing the version number in setup.py is far from the only thing you have to do to release a new version of a package. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Tue Jan 31 07:10:00 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 31 Jan 2012 06:10:00 +0000 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: <4F26618B.8030100@egenix.com> <5AFEE304CF20438284D88082D4D07044@gmail.com> Message-ID: <4F2785B8.70809@simplistix.co.uk> On 31/01/2012 05:26, Donald Stufft wrote: > The argument that bumping the release number is hard is an argument that > your process should include testing the package before releasing it, not > that it should open up anyone using PyPI to install packages for > potential damaging security issues on top of the possible random and > undefined breakage that can happen for seemingly no reason. Heh, Phil bumps his release numbers all the time, he just never bothers doing formal releases anymore ;-) It's always amusing watching the random selection of subversion revisions that come down when the latest setuptools is installed... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From fuzzyman at gmail.com Tue Jan 31 23:53:48 2012 From: fuzzyman at gmail.com (Michael Foord) Date: Tue, 31 Jan 2012 22:53:48 +0000 Subject: [Catalog-sig] Proposal: close the PyPI file-replacement loophole In-Reply-To: References: Message-ID: On 29 January 2012 23:47, Richard Jones wrote: > Hi catalog-sig, > > When we initially implemented file upload to PyPI it was our intention > that the file be immutable once uploaded. The goal was to make things > significantly simpler for end users - there would only ever be one > file with a given name. If the content changed then so must the name > (typically by creating a new release version.) > > After the upload facility was put in place we also added the ability > to delete files uploaded to pypi. This created a loophole: if a > package owner knew how to they could delete the file and re-upload, > thus circumventing the replacement protection. > > I'm considering closing this loophole by retaining a record of the > uploaded file (though not the contents) so that future uploads with > the same name wouldn't be allowed. I understand that this is how the > ruby gem archive handles deletion of files. > > Your thoughts? > FWIW I've occasionally found it useful to be able to delete uploads and replace them, so I'm -1 on losing this capability. All the best, Michael > > > Richard > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: