From richardjones at optushome.com.au Fri Jan 2 08:13:17 2009 From: richardjones at optushome.com.au (Richard Jones) Date: Fri, 2 Jan 2009 18:13:17 +1100 Subject: [Catalog-sig] Application downloads hosted by PyPI? Message-ID: <200901021813.17776.richardjones@optushome.com.au> When I initially implemented PyPI file hosting it was specifically designed to only handle files generated by distutils. I'm now in a position where I'd personally like to upload a application (well, a zip file of an application) to the index. I've solved the problem of generating the application distribution files* but to get it to upload I had to tell PyPI that it was an sdist, and include a PKG-INFO file (to pass the basic test I put in place to make sure that sdist files being uploaded really were sdist files). Clearly this is not optimal as the file is not really an sdist ;) I couldn't upload it as a "bdist" because PyPI understands that binary distributions are Python-version-specific. My application distribution is not version specific. What do people think about adding a new file type allowed for upload of "application" or similar? Richard * see my blog entry for more information: http://www.mechanicalcat.net/richard/log/Python/Sane_Python_application_packaging__initial_solution From jcea at jcea.es Mon Jan 5 18:03:29 2009 From: jcea at jcea.es (Jesus Cea) Date: Mon, 05 Jan 2009 18:03:29 +0100 Subject: [Catalog-sig] Replication and security Message-ID: <49623D61.70707@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Currently setuptools allows to upload a PGP signature along the package, to be able to check integrity and security. As far as I know, currently "easy_install" doesn't check it. That is bad, but life sucks. My problem now is with mirrors: How can anybody to validate files?. Beside the possible PGP signatures of authors (a check that should be integrated in "easy_install"), I would like PYPI main server (I guess it would be the single point where people upload new packages; the mirrors would be read-only) to digitally sign each uploaded package. This way, easy_install can check any package downloaded from any mirror, because PYPI public key would be a well known value. I have code in python to digitally sign/verify signatures using ElGamal algorithm. Any interest? - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSWI9YZlgi5GaxT1NAQLDFAQAjKWWmi9h3E4RvEupi03oAy839iCe7AO5 1nAHs+0aeQbQwskcUSD1RVZ4xP/AeJ+Gva1rvJfr7Ho41FD9WEFO/ErnHyGhEnL3 QK30lXbosnIWoqRiwXijrKtYp+9/pyixuDt7bL8hQ6ZBzgsOnknHaLJhDsNK+AMf KowdHXxsnPo= =eTrH -----END PGP SIGNATURE----- From martin at v.loewis.de Mon Jan 5 18:42:55 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 05 Jan 2009 18:42:55 +0100 Subject: [Catalog-sig] Replication and security In-Reply-To: <49623D61.70707@jcea.es> References: <49623D61.70707@jcea.es> Message-ID: <4962469F.6070505@v.loewis.de> > I have code in python to digitally sign/verify signatures using ElGamal > algorithm. Any interest? I rather prefer standard PGP signatures (with whatever signature algorithm the server key uses). Regards, Martin From jcea at jcea.es Mon Jan 5 18:56:36 2009 From: jcea at jcea.es (Jesus Cea) Date: Mon, 05 Jan 2009 18:56:36 +0100 Subject: [Catalog-sig] Replication and security In-Reply-To: <4962469F.6070505@v.loewis.de> References: <49623D61.70707@jcea.es> <4962469F.6070505@v.loewis.de> Message-ID: <496249D4.9060400@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >> I have code in python to digitally sign/verify signatures using ElGamal >> algorithm. Any interest? > > I rather prefer standard PGP signatures (with whatever signature > algorithm the server key uses). Me too, but then you requires an OpenPGP implementation in Python or a pgp/gpg program around, correctly configured, with the PYPI public key installed, etc. Instead, ElGamal signatures are verified in 12 lines of 100% python code. I am talking about checking that a package actually comes from PyPI, not the PGP author signature. This is important if anybody can deploy a mirror... At least "easy_install" can automatically verify that the downloaded package, from a mirror, was originated in the main PYPI server and it was not modified in any way. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSWJJ0Jlgi5GaxT1NAQKKMAP/QZGMLzVq1bBv3BU8HLTtMdygfH4CsH29 dYCxEcgbx7FmrfrdyN9cnAg9xnWR4S0u6ObnfhxVrx0+UyXivtdtTqDxh13TNJay 6U93QbILsrtr2Ey+yFDHg9VwmqNb9LMX/UUvBt2uyd1BEHbiKacPrqshTCyvhdHY aMW8rspseK4= =6/Hp -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Fri Jan 9 17:24:33 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 9 Jan 2009 17:24:33 +0100 Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <20090109154504.GA25799@fridge.pov.lt> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> Message-ID: <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> On Fri, Jan 9, 2009 at 4:45 PM, Marius Gedminas wrote: > On Fri, Jan 09, 2009 at 10:17:50AM -0500, Benji York wrote: >> On Fri, Jan 9, 2009 at 10:08 AM, Stephen Emslie wrote: >> > A bit OT, but from your blog post on the subject: >> > >> >>I'd like to go further and to think about a ssh-agent like system, so there's no need >> >>to enter the pasword everytime you work with PyPI in the same session. >> > >> > Have you had any feedback on this yet? >> >> Here's some: how about instead of an ssh-like system, use ssh itself. Front >> PyPI with an ssh server that users connect to. That way it is both secure and >> the infrastructure (agent, etc.) is already in place. > > Yes please. I'd rather have one agent running and reuse my SSH key for > authentication. That would be awesome indeed. But that would involve quite some changes on server side, I'll forward this mail to catalog-sig for Richard, Martin and others's feedback Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Fri Jan 9 21:18:20 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Jan 2009 21:18:20 +0100 Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> Message-ID: <4967B10C.6030904@v.loewis.de> >>> Here's some: how about instead of an ssh-like system, use ssh itself. Front >>> PyPI with an ssh server that users connect to. That way it is both secure and >>> the infrastructure (agent, etc.) is already in place. >> Yes please. I'd rather have one agent running and reuse my SSH key for >> authentication. > > That would be awesome indeed. But that would involve quite some > changes on server side, > I'll forward this mail to catalog-sig for Richard, Martin and others's feedback I'm fairly skeptical. First, the infrastructure is *not* yet in place. Nobody has uploaded SSH keys to PyPI, and in order to allow SSH access, we probably would need to create a Unix account, which then runs a fixed (Python) program on ssh login. That is much less secure than the current setup, in the sense that this program can probably tricked much easier than Apache can. So it opens a door for people hacking into the system; all they have to do is to create a fake PyPI account and upload an SSH key... To improve password storage, I think it would be better to use the platform's secure password storage services where available (e.g. OSX Keychain, KDE KWallet, etc). Of course, such a library should be developed independently of distutils. For Keychain, there is already http://muffinresearch.co.uk/archives/2008/02/05/python-keychainpy-access-to-the-mac-osx-keychain/ Regards, Martin From jim at zope.com Fri Jan 9 21:57:55 2009 From: jim at zope.com (Jim Fulton) Date: Fri, 9 Jan 2009 15:57:55 -0500 Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <4967B10C.6030904@v.loewis.de> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> Message-ID: <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com> On Jan 9, 2009, at 3:18 PM, Martin v. L?wis wrote: >>>> Here's some: how about instead of an ssh-like system, use ssh >>>> itself. Front >>>> PyPI with an ssh server that users connect to. That way it is >>>> both secure and >>>> the infrastructure (agent, etc.) is already in place. >>> Yes please. I'd rather have one agent running and reuse my SSH >>> key for >>> authentication. >> >> That would be awesome indeed. But that would involve quite some >> changes on server side, >> I'll forward this mail to catalog-sig for Richard, Martin and >> others's feedback > > I'm fairly skeptical. First, the infrastructure is *not* yet in place. > Nobody has uploaded SSH keys to PyPI, Right. PyPI would have to grow the ability to manage public keys for users. > and in order to allow SSH access, > we probably would need to create a Unix account, No, you would not. > which then runs a fixed > (Python) program on ssh login. That is much less secure than the > current > setup, in the sense that this program can probably tricked much easier > than Apache can. So it opens a door for people hacking into the > system; > all they have to do is to create a fake PyPI account and upload an SSH > key... No. You'd have a new server process, written in Python using Twisted or paramiko, that would would provide a small number of specialized commands and that would read public keys from the pypi database for authentication and update the database in response to commands, Jim -- Jim Fulton Zope Corporation From jim at zope.com Fri Jan 9 22:02:53 2009 From: jim at zope.com (Jim Fulton) Date: Fri, 9 Jan 2009 16:02:53 -0500 Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com> Message-ID: On Jan 9, 2009, at 3:57 PM, Jim Fulton wrote: > No. You'd have a new server process, written in Python using Twisted > or paramiko, that would would provide a small number of specialized > commands Or better yet, supported scp. Then the upload/register process would be reduced to just scp-ing a distro to pypi. The server could read meta-data from the distro, register the release, if necessary, and put the distro in the right place. Jim -- Jim Fulton Zope Corporation From martin at v.loewis.de Fri Jan 9 22:03:25 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Jan 2009 22:03:25 +0100 Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com> Message-ID: <4967BB9D.6070307@v.loewis.de> > No. You'd have a new server process, written in Python using Twisted or > paramiko, that would would provide a small number of specialized > commands and that would read public keys from the pypi database for > authentication and update the database in response to commands, Ok. I guess "contributions are welcome". Regards, Martin From martin at v.loewis.de Fri Jan 9 22:07:36 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Jan 2009 22:07:36 +0100 Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the password optional in .pypirc In-Reply-To: References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com> Message-ID: <4967BC98.8070508@v.loewis.de> > Or better yet, supported scp. Then the upload/register process would be > reduced to just scp-ing a distro to pypi. The server could read > meta-data from the distro, register the release, if necessary, and put > the distro in the right place. That wouldn't fit too well with the existing "register" and "upload" commands, I think. Regards, Martin From ziade.tarek at gmail.com Sat Jan 10 11:35:48 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 10 Jan 2009 11:35:48 +0100 Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <4967BC98.8070508@v.loewis.de> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com> <4967BC98.8070508@v.loewis.de> Message-ID: <94bdd2610901100235o6c1544b5u4e94a0fe6111304@mail.gmail.com> On Fri, Jan 9, 2009 at 10:07 PM, "Martin v. L?wis" wrote: >> Or better yet, supported scp. Then the upload/register process would be >> reduced to just scp-ing a distro to pypi. The server could read >> meta-data from the distro, register the release, if necessary, and put >> the distro in the right place. > > That wouldn't fit too well with the existing "register" and "upload" > commands, I think. +1 and in any case those commands should stay compatible with the current mechanism and let people store the password in the pypirc file if they want to, and use https authentication. Imho a scp/ssh protocol should be implemented in a new set of commands, Regards Tarek > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From tseaver at palladion.com Sun Jan 11 01:40:54 2009 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 10 Jan 2009 19:40:54 -0500 Subject: [Catalog-sig] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <4967B10C.6030904@v.loewis.de> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> Message-ID: <49694016.8080302@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >>>> Here's some: how about instead of an ssh-like system, use ssh itself. Front >>>> PyPI with an ssh server that users connect to. That way it is both secure and >>>> the infrastructure (agent, etc.) is already in place. >>> Yes please. I'd rather have one agent running and reuse my SSH key for >>> authentication. >> That would be awesome indeed. But that would involve quite some >> changes on server side, >> I'll forward this mail to catalog-sig for Richard, Martin and others's feedback > > I'm fairly skeptical. First, the infrastructure is *not* yet in place. > Nobody has uploaded SSH keys to PyPI, and in order to allow SSH access, > we probably would need to create a Unix account, which then runs a fixed > (Python) program on ssh login. Right, a single account with multiple keys (each with 'command='do_pypi - -u '). > That is much less secure than the current > setup, in the sense that this program can probably tricked much easier > than Apache can. So it opens a door for people hacking into the system; > all they have to do is to create a fake PyPI account and upload an SSH > key... Zope has been using the 'command=' bit to do SSH-protected CVS / SVN access since 2000 with a lot of success; 370+ committers have keys on the system. The command being executed is actually a small shell script, which barfs if the program being run is not one of 'svn', 'cvs', or 'scp' (for uploading tarballs). > To improve password storage, I think it would be better to use the > platform's secure password storage services where available (e.g. > OSX Keychain, KDE KWallet, etc). Of course, such a library should be > developed independently of distutils. For Keychain, there is already > > http://muffinresearch.co.uk/archives/2008/02/05/python-keychainpy-access-to-the-mac-osx-keychain/ Not only are PyPI passwords stored in the clear on user's hard drives, they are sent in the clear on every authenticated request to the web interface (basic auth over unencrypted HTTP): it seems to me we ought to worry about both those issues more. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJaUAW+gerLs4ltQ4RAhFXAJ47WOzMAe12m+YD5BNu22BzTU+QRQCeLTbX DSaVk1I96K5mzaZro98HUTU= =8sRs -----END PGP SIGNATURE----- From martin at v.loewis.de Sun Jan 11 04:35:36 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 11 Jan 2009 04:35:36 +0100 Subject: [Catalog-sig] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <49694016.8080302@palladion.com> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> <49694016.8080302@palladion.com> Message-ID: <49696908.1070501@v.loewis.de> >> That is much less secure than the current >> setup, in the sense that this program can probably tricked much easier >> than Apache can. So it opens a door for people hacking into the system; >> all they have to do is to create a fake PyPI account and upload an SSH >> key... > > Zope has been using the 'command=' bit to do SSH-protected CVS / SVN > access since 2000 with a lot of success; 370+ committers have keys on > the system. The command being executed is actually a small shell > script, which barfs if the program being run is not one of 'svn', 'cvs', > or 'scp' (for uploading tarballs). Well, then good luck that nobody has tried to hack your script. E.g. might it work that I somehow manage to upload a svn binary onto your system (e.g. by first checking it in, and relying on an automated checkout process that runs somewhere), then invoke this binary through the shell account? > Not only are PyPI passwords stored in the clear on user's hard drives, > they are sent in the clear on every authenticated request to the web > interface (basic auth over unencrypted HTTP): it seems to me we ought > to worry about both those issues more. Perhaps. Contributions are welcome. Regards, Martin From ziade.tarek at gmail.com Sun Jan 11 10:29:21 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 11 Jan 2009 10:29:21 +0100 Subject: [Catalog-sig] [distutils] make the storage of the password optional in .pypirc In-Reply-To: <49696908.1070501@v.loewis.de> References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com> <4967B10C.6030904@v.loewis.de> <49694016.8080302@palladion.com> <49696908.1070501@v.loewis.de> Message-ID: <94bdd2610901110129q545346c9ka41fa9319523ee89@mail.gmail.com> On Sun, Jan 11, 2009 at 4:35 AM, "Martin v. L?wis" wrote: >> Not only are PyPI passwords stored in the clear on user's hard drives, >> they are sent in the clear on every authenticated request to the web >> interface (basic auth over unencrypted HTTP): it seems to me we ought >> to worry about both those issues more. > > Perhaps. Contributions are welcome. Can we finish on the PyPI mirroring contribution before we start this one ? (since you are our entry point Martin on these topics) I have finished my tests on my side. And I have a branch ready here https://svn.python.org/packages/branches/tarek-pypi/pypi/ I would like to make more tests with a realistic flow of data, and I am waiting for some feedback/help on this work. here's how we could proceed: phase 1 : proving non-regression 1 - I need an access to the pypi log files produced by Apache (a simple browsable view of the log directory should be enough and not risky) 2 - on my side I can grab those files daily right and put them on my PyPI server instance, and run the process like if I was on the real server. 3 - I will make this version reachable on my server, so we can check that there's no regression = the count of the package that existed before the dump I had should be equal and grow the same way on both sides. phase 2 - testing the mirroring 4 - I will maintain a fake "mirror" that will be registered and will provide realistic stats (a copy of the pypi apache log, where I will keep just one hit per package file) 5 - we will validate that the global-stats and local-stats files generated are right, and that the counts are the sum of pypi and the mirror. (pypi+1) If we can do that before Pycon maybe Pycon sprints could be the place where we launch the mirroring, and start the SSH project if Jean-Paul and others are willing to jump in ? Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From rhijnauwen at gmail.com Wed Jan 14 16:06:59 2009 From: rhijnauwen at gmail.com (Bart Spaans) Date: Wed, 14 Jan 2009 16:06:59 +0100 Subject: [Catalog-sig] How do we change project ownership? Message-ID: <881c57460901140706x2376ef97n582945226128d96b@mail.gmail.com> Hi, I would like to take over the maintenance of a piece of software (pyFluidSynth), but we can't seem to find a 'change owner' or 'add owner' option. Is it possible to change the owner of a certain package or should the original owner completely delete the package first? If so, will that free up the name? Best regards, Bart Spaans. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at zopyx.com Wed Jan 14 16:17:59 2009 From: lists at zopyx.com (Andreas Jung) Date: Wed, 14 Jan 2009 16:17:59 +0100 Subject: [Catalog-sig] How do we change project ownership? In-Reply-To: <881c57460901140706x2376ef97n582945226128d96b@mail.gmail.com> References: <881c57460901140706x2376ef97n582945226128d96b@mail.gmail.com> Message-ID: <496E0227.5050507@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 14.01.2009 16:06 Uhr, Bart Spaans wrote: > Hi, > > I would like to take over the maintenance of a piece of software > (pyFluidSynth), but we can't seem to find a 'change owner' or 'add > owner' option. Is it possible to change the owner of a certain package > or should the original owner completely delete the package first? If so, > will that free up the name? > Please look carefully. There is link "Administer the ROle assigned to users for this package" after having logged and choosing a package belonging to you. - -aj -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkluAicACgkQCJIWIbr9KYzXlQCcDeIIWee6IHtkaOfyh/vKSPuy tQsAnRTJujPkAoBLgo7mR4wUjjbxRozu =6Rx+ -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From martin at v.loewis.de Sat Jan 17 15:45:07 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 17 Jan 2009 15:45:07 +0100 Subject: [Catalog-sig] New log record: rename Message-ID: <4971EEF3.1030406@v.loewis.de> Those of you monitoring the changelog might notice a new log record, for when a package gets renamed. There is currently no UI for renaming packages, so this is really restricted to the PyPI administrator (renaming requests can be submitted to the PyPI bug tracker). Regards, Martin From martin at v.loewis.de Sat Jan 17 15:47:47 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 17 Jan 2009 15:47:47 +0100 Subject: [Catalog-sig] Automatic redirects for normalized names Message-ID: <4971EF93.8020704@v.loewis.de> Now that PyPI is free of name collisions with respect to the (setuptools) normalized_name, the "simple" API offers redirects in cases where a name was misspelled. E.g. accessing http://pypi.python.org/simple/pyxml will redirect to http://pypi.python.org/simple/PyXML This should remove the need to download the entire simple index in setuptools, in most cases. Regards, Martin From jcea at jcea.es Wed Jan 21 17:19:32 2009 From: jcea at jcea.es (Jesus Cea) Date: Wed, 21 Jan 2009 17:19:32 +0100 Subject: [Catalog-sig] Can not submit new packages Message-ID: <49774B14.90109@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I am trying to upload a new package to PYPI, and the sending is failing with an authentication failure. I already register again with "python setup.py register" (using my old credentials), and the server gives "200 OK".. But trying "python setup.py sdist upload --sign --show-response" gives this error: Submitting dist/bsddb3-4.7.4.tar.gz to http://pypi.python.org/pypi Upload failed (401): You must be identified to edit package information - --------------------------------------------------------------------------- Login required

You must be identified to edit package information

If you are a new user, please register.

If you have forgotten your password, you can have it reset for you.

- --------------------------------------------------------------------------- - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSXdLD5lgi5GaxT1NAQKFsgP/R3djp/2pwEsGJ2T0KTqv2xDEEaRzZLkb A4agXQqQ6VXM1Zd5KPR+z+/jjUAEIsb/Glih6QIrMS2dyNbE4C8w9i1ktqOCHpjQ h78mbzoWwJ9GaEwlb1vZIjazFdewIYyCsAXjZaB3VGiHSOStGUiPXG1X70eAyujo VWoSqV/GIbs= =Bi7t -----END PGP SIGNATURE----- From martin at v.loewis.de Wed Jan 21 20:53:08 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 21 Jan 2009 20:53:08 +0100 Subject: [Catalog-sig] Can not submit new packages In-Reply-To: <49774B14.90109@jcea.es> References: <49774B14.90109@jcea.es> Message-ID: <49777D24.7080003@v.loewis.de> > Submitting dist/bsddb3-4.7.4.tar.gz to http://pypi.python.org/pypi > Upload failed (401): You must be identified to edit package information Can you debug this further to find out what is really happening? Regards, Martin From jcea at jcea.es Wed Jan 21 21:38:04 2009 From: jcea at jcea.es (Jesus Cea) Date: Wed, 21 Jan 2009 21:38:04 +0100 Subject: [Catalog-sig] Can not submit new packages In-Reply-To: <49777D24.7080003@v.loewis.de> References: <49774B14.90109@jcea.es> <49777D24.7080003@v.loewis.de> Message-ID: <497787AC.70804@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >> Submitting dist/bsddb3-4.7.4.tar.gz to http://pypi.python.org/pypi >> Upload failed (401): You must be identified to edit package information > > Can you debug this further to find out what is really happening? Too late. I uploaded the packages manually via the webpage. It worked fine. Sorry, the upload was urgent. Hope this issue is still alive and somebody else hits it :). - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSXeHpplgi5GaxT1NAQJpBwP/QoilGvtklvFVfnQCN58CdKgSGjCxdxw+ u0uAD3/19b3ZNUeyC4Zd4KlJsIp6FRI37XBapg/7+oQF65T4QqwIh+iBIrsY86r4 m5dRNsIBa8zvv9ZzHPe33Ekne5bdVFPr4vF9Lx+ktX1FpMvV/KeukPkk5ZcY7DKT JIv6kj/IWr4= =wEBA -----END PGP SIGNATURE----- From jcea at jcea.es Wed Jan 21 21:40:34 2009 From: jcea at jcea.es (Jesus Cea) Date: Wed, 21 Jan 2009 21:40:34 +0100 Subject: [Catalog-sig] Can not submit new packages In-Reply-To: <497787AC.70804@jcea.es> References: <49774B14.90109@jcea.es> <49777D24.7080003@v.loewis.de> <497787AC.70804@jcea.es> Message-ID: <49778842.6050705@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jesus Cea wrote: > Hope this issue is still alive and somebody else hits it :). The strange thing is that I had enough privileges to create the release (4.7.4), but the file upload failed. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSXeIQplgi5GaxT1NAQIVpQP/dBMgZ3HwNNXiGZoKsEK2f0cuyxByHsEl 9d716pS76ARsIJpkovuV4vX9uD54p52Gp74H6BiMrf9dnFP7W8wJmxrrHBqepbF8 JWxHIHma3FerAuSrh0xilmuM2l4McZ3AzlFPp91Ny0wkpNHdErnNKz4jYWbCr8mg TsdGM0nlA+g= =GzIm -----END PGP SIGNATURE----- From szybalski at gmail.com Thu Jan 22 05:52:53 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Wed, 21 Jan 2009 22:52:53 -0600 Subject: [Catalog-sig] local copy of pypi packages list, and package data, how? Message-ID: <804e5c70901212052j2e9eb544o102e2b929e2441db@mail.gmail.com> Hello, I've been looking into xmlrpc interface that you have for pypi. I am able to browse all packages and get the data about them...keywords,etc....I want to have a local version of the catalog data and keep it in sync daily. What I'm wondering is how can I keep my app in sync? Here is what I'm doing right now, I was wondering if this is not overloading your servers, or is there a faster/more efficient way. 1. I get a list of all 5000+ packages. 2. For each package I get a version number. 3. For each (package,version#) I get the package data. 4. Sync daily using updated_releases I use the package data to look for certain keywords. Process 2 seem to take around 20+minutes, process 3 takes more, but after first time I can just get the new updated packages since the last time and run these. 1. Is there a xmlrpc function that I can use to search for keywords and just get the packages I need? 2. Is there a better strategy then what I am doing? I would like to sync daily. http://lucasmanual.com/blog/2009/how-to-get-information-from-pypi-via-xmlrpc/ Thanks, Lucas -- How to create python package? http://lucasmanual.com/mywiki/PythonPaste Bazaar and Launchpad http://lucasmanual.com/mywiki/Bazaar From amk at amk.ca Thu Jan 22 13:02:36 2009 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 22 Jan 2009 07:02:36 -0500 Subject: [Catalog-sig] PyPI load this morning Message-ID: <20090122120236.GA8545@amk.local> Some sort of script was hitting the top page on PyPI this morning, driving the machine's load average to 15. I added: # 2009-01-22 deny from 213.41.97.133 to the cheeseshop config and reloaded Apache, and the load average has now dropped to around .3. --amk From lists at zopyx.com Fri Jan 23 15:09:31 2009 From: lists at zopyx.com (Andreas Jung) Date: Fri, 23 Jan 2009 15:09:31 +0100 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index Message-ID: <4979CF9B.1080001@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, the simplex index also contains stuff belonging to hidden releases. Is this intentional? Example (threadframe package): http://pypi.python.org/pypi/threadframe The XMLRPC API reveals only one published version: 0.2 The simple index contains also a 1.0 release which is hidden: http://pypi.python.org/simple/threadframe I suggest that the simple index should only show stuff belong to un-hidden releases. Andreas - -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK - ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkl5z5sACgkQCJIWIbr9KYxbxQCfZsJ7jfBWA4tjBA/uG0/4CbhP NkcAn0Guz5UvZj6axswJBwlOieJAzOS4 =YoCs -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From fdrake at gmail.com Fri Jan 23 15:15:47 2009 From: fdrake at gmail.com (Fred Drake) Date: Fri, 23 Jan 2009 09:15:47 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <4979CF9B.1080001@zopyx.com> References: <4979CF9B.1080001@zopyx.com> Message-ID: <9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com> On Fri, Jan 23, 2009 at 9:09 AM, Andreas Jung wrote: > the simplex index also contains stuff belonging to hidden releases. > Is this intentional? Yes. Projects that are already using a version still need to be able to find the releases; this is common for projects that specify exact versions for deployment purposes. The simple index is the right place for this; tools like setuptools and zc.buildout use the simple index for automated operations by default. The web UI doesn't show hidden releases, which is fine for interactive users. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From benji at benjiyork.com Fri Jan 23 16:11:53 2009 From: benji at benjiyork.com (Benji York) Date: Fri, 23 Jan 2009 10:11:53 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com> References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com> Message-ID: On Fri, Jan 23, 2009 at 9:15 AM, Fred Drake wrote: > On Fri, Jan 23, 2009 at 9:09 AM, Andreas Jung wrote: >> the simplex index also contains stuff belonging to hidden releases. >> Is this intentional? > > Yes. > > Projects that are already using a version still need to be able to > find the releases; this is common for projects that specify exact > versions for deployment purposes. Exactly. > The web UI doesn't show hidden releases, which is fine for interactive users. As an interactive user, I've never understood why the hidden release functionality exists; especially automatically hiding old releases. -- Benji York From fdrake at gmail.com Fri Jan 23 16:14:01 2009 From: fdrake at gmail.com (Fred Drake) Date: Fri, 23 Jan 2009 10:14:01 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com> Message-ID: <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> On Fri, Jan 23, 2009 at 10:11 AM, Benji York wrote: > As an interactive user, I've never understood why the hidden release > functionality exists; especially automatically hiding old releases. I can only guess at the original motivations myself, and that's not really helpful. The automatic hiding of older releases is definitely a mistake; I've seen no good come of it, and it's surprising. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From richardjones at optushome.com.au Sat Jan 24 02:44:30 2009 From: richardjones at optushome.com.au (Richard Jones) Date: Sat, 24 Jan 2009 12:44:30 +1100 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> Message-ID: <200901241244.30479.richardjones@optushome.com.au> On Sat, 24 Jan 2009, Fred Drake wrote: > On Fri, Jan 23, 2009 at 10:11 AM, Benji York wrote: > > As an interactive user, I've never understood why the hidden release > > functionality exists; especially automatically hiding old releases. > > I can only guess at the original motivations myself, and that's not > really helpful. It seemed like a good idea at the time :) > The automatic hiding of older releases is definitely a mistake; I've > seen no good come of it, and it's surprising. Proposals to change it are, as with everything PyPI, welcome :) Richard From jim at zope.com Sat Jan 24 16:57:48 2009 From: jim at zope.com (Jim Fulton) Date: Sat, 24 Jan 2009 10:57:48 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <200901241244.30479.richardjones@optushome.com.au> References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> <200901241244.30479.richardjones@optushome.com.au> Message-ID: On Jan 23, 2009, at 8:44 PM, Richard Jones wrote: ... >> The automatic hiding of older releases is definitely a mistake; I've >> seen no good come of it, and it's surprising. > > Proposals to change it are, as with everything PyPI, welcome :) What about: http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html :) Jim -- Jim Fulton Zope Corporation From fdrake at gmail.com Sat Jan 24 17:27:41 2009 From: fdrake at gmail.com (Fred Drake) Date: Sat, 24 Jan 2009 11:27:41 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> <200901241244.30479.richardjones@optushome.com.au> Message-ID: <9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com> On Sat, Jan 24, 2009 at 10:57 AM, Jim Fulton wrote: > What about: > > http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html I think Richard and Martin are waiting for the patch I've never had time to look into. :-( -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From martin at v.loewis.de Sat Jan 24 20:34:02 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Jan 2009 20:34:02 +0100 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com> References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> <200901241244.30479.richardjones@optushome.com.au> <9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com> Message-ID: <497B6D2A.9080905@v.loewis.de> Fred Drake wrote: > On Sat, Jan 24, 2009 at 10:57 AM, Jim Fulton wrote: >> What about: >> >> http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html > > I think Richard and Martin are waiting for the patch I've never had > time to look into. :-( It's a bit different: in absence of a patch, I had simply forgotten about it (IOW, I wasn't waiting). I have now implemented that feature; there is a checkbox on each package telling whether old revisions should automatically get hidden. Regards, Martin From martin at v.loewis.de Sat Jan 24 20:36:29 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Jan 2009 20:36:29 +0100 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <4979CF9B.1080001@zopyx.com> References: <4979CF9B.1080001@zopyx.com> Message-ID: <497B6DBD.2080409@v.loewis.de> > the simplex index also contains stuff belonging to hidden releases. > Is this intentional? As others have already explained: yes. It was made the way it is specifically on request of setuptools users (as was the entire /simple) index. > I suggest that the simple index should only show stuff belong to > un-hidden releases. Chances of changing PyPI would be slightly (but not much) higher if you explained *why* you want this changed. Regards, Martin From martin at v.loewis.de Sat Jan 24 20:48:52 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Jan 2009 20:48:52 +0100 Subject: [Catalog-sig] local copy of pypi packages list, and package data, how? In-Reply-To: <804e5c70901212052j2e9eb544o102e2b929e2441db@mail.gmail.com> References: <804e5c70901212052j2e9eb544o102e2b929e2441db@mail.gmail.com> Message-ID: <497B70A4.8010009@v.loewis.de> > What I'm wondering is how can I keep my app in sync? There are a number of PyPI mirroring solutions out there; I suggest you use one of them: http://pypi.python.org/pypi/z3c.pypimirror https://launchpad.net/~pypi-mirror > Here is what I'm > doing right now, I was wondering if this is not overloading your > servers, or is there a faster/more efficient way. See AMK's recent message - perhaps it was you who was overloading the server. > 1. Is there a xmlrpc function that I can use to search for keywords > and just get the packages I need? No; you might use the regular UI search function, of course, but please do restrict this to a small number of queries per hour. > 2. Is there a better strategy then what I am doing? I would like to sync daily. For downloading all files for a package, you might want to use the simple API (/simple). For keeping in sync, you might want to use changelog; updated_releases will only tell you whether a new release was made, not whether a file has been added or replaced. Regards, Martin From fdrake at gmail.com Sat Jan 24 20:49:00 2009 From: fdrake at gmail.com (Fred Drake) Date: Sat, 24 Jan 2009 14:49:00 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <497B6D2A.9080905@v.loewis.de> References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> <200901241244.30479.richardjones@optushome.com.au> <9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com> <497B6D2A.9080905@v.loewis.de> Message-ID: <9cee7ab80901241149w4f655987t15ca4fc0a32fa84c@mail.gmail.com> On Sat, Jan 24, 2009 at 2:34 PM, "Martin v. L?wis" wrote: > I have now implemented that feature; there is a checkbox on each > package telling whether old revisions should automatically get hidden. Wonderful! This is really nice; I've started switching projects I manage that should have this toggled. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From benji at benjiyork.com Sat Jan 24 21:24:30 2009 From: benji at benjiyork.com (Benji York) Date: Sat, 24 Jan 2009 15:24:30 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <497B6D2A.9080905@v.loewis.de> References: <4979CF9B.1080001@zopyx.com> <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> <200901241244.30479.richardjones@optushome.com.au> <9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com> <497B6D2A.9080905@v.loewis.de> Message-ID: On Sat, Jan 24, 2009 at 2:34 PM, "Martin v. L?wis" wrote: > Fred Drake wrote: >> On Sat, Jan 24, 2009 at 10:57 AM, Jim Fulton wrote: >>> What about: >>> >>> http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html >> >> I think Richard and Martin are waiting for the patch I've never had >> time to look into. :-( > > It's a bit different: in absence of a patch, I had simply forgotten > about it (IOW, I wasn't waiting). > > I have now implemented that feature; there is a checkbox on each > package telling whether old revisions should automatically get hidden. Great! -- Benji York From lists at zopyx.com Sun Jan 25 12:12:51 2009 From: lists at zopyx.com (Andreas Jung) Date: Sun, 25 Jan 2009 12:12:51 +0100 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <4979CF9B.1080001@zopyx.com> References: <4979CF9B.1080001@zopyx.com> Message-ID: <497C4933.1020400@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Also related to my initial question: how to deal with broken package releases if the package maintainer(s) don't response within a reasonable timeframe? My particular case: the 1.0 release of 'threadframe' is hidden but visible within the simple index. This 1.0 release is broken since the download URL actually points to the 0.1 release of the package. This is highly confusing in the context of zc.buildout and PyPI mirror because the 1.0 release appears as being the most current version however no threadframe-1.0* package is available from the download page. Andreas On 23.01.2009 15:09 Uhr, Andreas Jung wrote: > Hi there, > > the simplex index also contains stuff belonging to hidden releases. > Is this intentional? > > Example (threadframe package): > > http://pypi.python.org/pypi/threadframe > > The XMLRPC API reveals only one published version: 0.2 > > The simple index contains also a 1.0 release which is hidden: > > http://pypi.python.org/simple/threadframe > > I suggest that the simple index should only show stuff belong to > un-hidden releases. > > Andreas > - ------------------------------------------------------------------------ _______________________________________________ Catalog-SIG mailing list Catalog-SIG at python.org http://mail.python.org/mailman/listinfo/catalog-sig - -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK - ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkl8STMACgkQCJIWIbr9KYwa/wCgjhc1TnF1nGDuFnweNjpXbbyU qJ8An3DuiEyok942mKORKFV5WLEyr0aI =uUh1 -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From lists at zopyx.com Sun Jan 25 13:50:44 2009 From: lists at zopyx.com (Andreas Jung) Date: Sun, 25 Jan 2009 13:50:44 +0100 Subject: [Catalog-sig] [PyPI] Creation date of releases and release files? Message-ID: <497C6024.5090509@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Would it take much effort including the creation date of packages (and their release files) on PyPI? Andreas - -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK - ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkl8YCQACgkQCJIWIbr9KYwLGgCdHkNNG0DXnDwv97D8ioc6OvN+ hggAn2k6zNTj4O+bYWCsL3ukLexMR3z/ =ErMO -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From tarek.ziade at ingeniweb.com Sun Jan 25 14:05:28 2009 From: tarek.ziade at ingeniweb.com (Tarek Ziade) Date: Sun, 25 Jan 2009 14:05:28 +0100 Subject: [Catalog-sig] [PyPI] Creation date of releases and release files? In-Reply-To: <497C6024.5090509@zopyx.com> References: <497C6024.5090509@zopyx.com> Message-ID: 2009/1/25 Andreas Jung : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Would it take much effort including the creation date of packages (and > their release files) on PyPI? I guess not because the date is already stored in the database in the "journals" table, so this would just require to change the UI to display it. But maybe it would be better to include a date field in the "releases" table imho to avoid an extra query join when displaying release infos. Tarek -- Tarek Ziad? - Directeur Technique INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632 Bureaux de la Colline - 1 rue Royale - B?timent D - 9?me ?tage 92210 Saint Cloud - France Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04 http://www.ingeniweb.com - une soci?t? du groupe Alter Way From lists at zopyx.com Sun Jan 25 14:19:38 2009 From: lists at zopyx.com (Andreas Jung) Date: Sun, 25 Jan 2009 14:19:38 +0100 Subject: [Catalog-sig] [zc.buildout] Dealing with building (large) libraries Message-ID: <497C66EA.3070109@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, it had become a common pattern with buildout compiling almost all and everything within one buildout. E.g. the Deliverance integration using plone.recipe.deliverance downloads and compiles libxml2/libxslt which takes a lot of time. In addition we have seen unmotivated uninstall/install orgies of parts (possibly the related recipes are to blame) causing a lot of turnaround time for developers (and frustration about using buildout). Anyone having similar experiences and/or hints how deal with such larger buildouts? We are having a company internal sprint next week where we are thinking about a 2-stage buildout for some of our projects where the fat parts will be moved to a dedicated buildout configuration and installed/maintained as as global resources. This will at least reduce the number of pointless uninstall/install cycles. Andreas - -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK - ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkl8ZuoACgkQCJIWIbr9KYwQ6QCg5IuQ8aw+tIp7rgfBnT45A7VK b1QAoOoFv9w0w+iYOMWtUR4BzZ2t0Qad =bxE1 -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From jim at zope.com Sun Jan 25 17:42:37 2009 From: jim at zope.com (Jim Fulton) Date: Sun, 25 Jan 2009 11:42:37 -0500 Subject: [Catalog-sig] [zc.buildout] Dealing with building (large) libraries In-Reply-To: <497C66EA.3070109@zopyx.com> References: <497C66EA.3070109@zopyx.com> Message-ID: <5EF42A12-D48F-4772-A51A-CFEF93AE1A16@zope.com> On Jan 25, 2009, at 8:19 AM, Andreas Jung wrote: > it had become a common pattern with buildout compiling almost all and > everything within one buildout. E.g. the Deliverance integration using > plone.recipe.deliverance downloads and compiles libxml2/libxslt which > takes a lot of time. In addition we have seen unmotivated > uninstall/install orgies of parts (possibly the related recipes are to > blame) causing a lot of turnaround time for developers (and > frustration > about using buildout). Buildout takes a conservative approach when deciding whether a part needs to be reinstalled. In particular, a change to a part's recipe (like a new recipe egg) or a package the recipe depends on (e.g. buildout itself) will cause a part to be reinstalled. > Anyone having similar experiences and/or hints how deal with such > larger > buildouts? I don't think it's really a question of the size of the buildout so much as the expense of individual parts. Many or most parts aren't expensive to reinstall. Certain parts, like those that build a big external library can be especially painful. > We are having a company internal sprint next week where we > are thinking about a 2-stage buildout for some of our projects where > the > fat parts will be moved to a dedicated buildout configuration and > installed/maintained as as global resources. This will at least reduce > the number of pointless uninstall/install cycles. That's a reasonable approach. Another approach might be to add an option to make buildout less conservative about certain parts. For example, there might be an option to, for a given list of parts to only reinstall a part if an option changes, ignoring changes to the version of the part recipe or it's dependencies. Alternatively, we could change buildout to use a provided value __buildout_signature__, rather than computing one itself if the option is provided. Then, for expensive parts, like one building a library, once could simply provide this option, giving a static value. I think this would be more effective that managing separate buildouts to compute expensive parts. Jim -- Jim Fulton Zope Corporation From jim at zope.com Sun Jan 25 17:46:30 2009 From: jim at zope.com (Jim Fulton) Date: Sun, 25 Jan 2009 11:46:30 -0500 Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index In-Reply-To: <497C4933.1020400@zopyx.com> References: <4979CF9B.1080001@zopyx.com> <497C4933.1020400@zopyx.com> Message-ID: <6CF36ABE-8CCC-4A6F-9F9C-08DF5FAD5263@zope.com> On Jan 25, 2009, at 6:12 AM, Andreas Jung wrote: > how to deal with broken package releases if the package maintainer(s) > don't response within a reasonable timeframe? My particular case: > the 1.0 release of 'threadframe' is hidden but visible within the > simple > index. This 1.0 release is broken since the download URL actually > points > to the 0.1 release of the package. This is highly confusing in the > context of zc.buildout and PyPI mirror because the 1.0 release appears > as being the most current version however no threadframe-1.0* > package is > available from the download page. To whom is this confusing? I don't see how buildout or setuptools would be confused by this situation. All they ultimately care about is the actual distributions they find. Jim -- Jim Fulton Zope Corporation From martin at v.loewis.de Sun Jan 25 19:20:16 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Jan 2009 19:20:16 +0100 Subject: [Catalog-sig] [PyPI] Creation date of releases and release files? In-Reply-To: <497C6024.5090509@zopyx.com> References: <497C6024.5090509@zopyx.com> Message-ID: <497CAD60.5060300@v.loewis.de> Andreas Jung wrote: > Would it take much effort including the creation date of packages (and > their release files) on PyPI? As Tarek says: the creation date is already available from the changelog. The upload dates are available from the changelog also, but more easily so directly from the file system. So I don't feel inclined to do anything about this - if you need it, the information is there. If you want to have it available more conveniently, contribute patches. Regards, Martin From lists at zopyx.com Sun Jan 25 19:42:24 2009 From: lists at zopyx.com (Andreas Jung) Date: Sun, 25 Jan 2009 19:42:24 +0100 Subject: [Catalog-sig] [PyPI] Creation date of releases and release files? In-Reply-To: <497CAD60.5060300@v.loewis.de> References: <497C6024.5090509@zopyx.com> <497CAD60.5060300@v.loewis.de> Message-ID: <497CB290.7050908@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 25.01.2009 19:20 Uhr, Martin v. L?wis wrote: > Andreas Jung wrote: >> Would it take much effort including the creation date of packages (and >> their release files) on PyPI? > > As Tarek says: the creation date is already available from the > changelog. The upload dates are available from the changelog also, > but more easily so directly from the file system. > > So I don't feel inclined to do anything about this - if you need it, the > information is there. If you want to have it available more > conveniently, contribute patches. Huh? The change is likely a one-liner within the related template code or whatever is behind the PyPI web UI - but anyway... - -aj -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkl8spAACgkQCJIWIbr9KYx7HwCeM3UD3Dafi2+YhVVdgJyDBU/2 32cAoMr0OGxFQ5jdgso3fErCRAKOSTKo =YMW+ -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From martin at v.loewis.de Sun Jan 25 20:24:41 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 25 Jan 2009 20:24:41 +0100 Subject: [Catalog-sig] [PyPI] Creation date of releases and release files? In-Reply-To: <497CB290.7050908@zopyx.com> References: <497C6024.5090509@zopyx.com> <497CAD60.5060300@v.loewis.de> <497CB290.7050908@zopyx.com> Message-ID: <497CBC79.3070105@v.loewis.de> > Huh? The change is likely a one-liner within the related template code > or whatever is behind the PyPI web UI - but anyway... The last change took me several our, because I had to fight ZPT; I try to reduce that to a minimum. Again, patches welcome. Regards, Martin From adam.boduch at gmail.com Wed Jan 28 15:01:12 2009 From: adam.boduch at gmail.com (Adam Boduch) Date: Wed, 28 Jan 2009 09:01:12 -0500 Subject: [Catalog-sig] API Documentation Message-ID: <1233151272.6653.3.camel@adam-laptop> Hi, I'm just wondering how to use the hosted API documentation feature on pypi. I followed the instructions for uploading a zip file containing the documentation. I now get HTTP forbidden when visiting http://packages.python.org/boduch/ Am I missing a step? Thanks in advance, Adam From martin at v.loewis.de Wed Jan 28 18:44:10 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 28 Jan 2009 18:44:10 +0100 Subject: [Catalog-sig] API Documentation In-Reply-To: <1233151272.6653.3.camel@adam-laptop> References: <1233151272.6653.3.camel@adam-laptop> Message-ID: <4980996A.7030702@v.loewis.de> Adam Boduch wrote: > Hi, > > I'm just wondering how to use the hosted API documentation feature on > pypi. I followed the instructions for uploading a zip file containing > the documentation. I now get HTTP forbidden when visiting > http://packages.python.org/boduch/ > > Am I missing a step? No - you just have the URL wrong. It's http://packages.python.org/boduch/boduch/ (apparently, your zipfile contains a single directory "boduch", and no index.html beside it) Regards, Martin From adam.boduch at gmail.com Wed Jan 28 19:10:49 2009 From: adam.boduch at gmail.com (Adam Boduch) Date: Wed, 28 Jan 2009 13:10:49 -0500 Subject: [Catalog-sig] API Documentation In-Reply-To: <4980996A.7030702@v.loewis.de> References: <1233151272.6653.3.camel@adam-laptop> <4980996A.7030702@v.loewis.de> Message-ID: <1233166249.6489.0.camel@adam-laptop> Excellent. Thanks so much! On Wed, 2009-01-28 at 18:44 +0100, "Martin v. L?wis" wrote: > Adam Boduch wrote: > > Hi, > > > > I'm just wondering how to use the hosted API documentation feature on > > pypi. I followed the instructions for uploading a zip file containing > > the documentation. I now get HTTP forbidden when visiting > > http://packages.python.org/boduch/ > > > > Am I missing a step? > > No - you just have the URL wrong. It's > > http://packages.python.org/boduch/boduch/ > > (apparently, your zipfile contains a single directory "boduch", and no > index.html beside it) > > Regards, > Martin From martin at v.loewis.de Wed Jan 28 19:15:33 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 28 Jan 2009 19:15:33 +0100 Subject: [Catalog-sig] API Documentation In-Reply-To: <1233166249.6489.0.camel@adam-laptop> References: <1233151272.6653.3.camel@adam-laptop> <4980996A.7030702@v.loewis.de> <1233166249.6489.0.camel@adam-laptop> Message-ID: <4980A0C5.6000800@v.loewis.de> Adam Boduch wrote: > Excellent. Thanks so much! You are welcome. I do recommend that you change the zipfile, though, to remove one URL level. Regards, Martin From szybalski at gmail.com Wed Jan 28 19:55:36 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Wed, 28 Jan 2009 12:55:36 -0600 Subject: [Catalog-sig] API Documentation In-Reply-To: <4980A0C5.6000800@v.loewis.de> References: <1233151272.6653.3.camel@adam-laptop> <4980996A.7030702@v.loewis.de> <1233166249.6489.0.camel@adam-laptop> <4980A0C5.6000800@v.loewis.de> Message-ID: <804e5c70901281055w64020000l78fbaa4086dbbb81@mail.gmail.com> How do you upload the documentation? Is there a link somewhere that has instructions on getting these docs in. I might use it for my package. Thanks, Lucas On Wed, Jan 28, 2009 at 12:15 PM, "Martin v. L?wis" wrote: > Adam Boduch wrote: >> Excellent. Thanks so much! > > You are welcome. I do recommend that you change the zipfile, though, to > remove one URL level. > > Regards, > Martin > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- How to create python package? http://lucasmanual.com/mywiki/PythonPaste Bazaar and Launchpad http://lucasmanual.com/mywiki/Bazaar From adam.boduch at gmail.com Wed Jan 28 20:12:31 2009 From: adam.boduch at gmail.com (Adam Boduch) Date: Wed, 28 Jan 2009 14:12:31 -0500 Subject: [Catalog-sig] API Documentation In-Reply-To: <4980A0C5.6000800@v.loewis.de> References: <1233151272.6653.3.camel@adam-laptop> <4980996A.7030702@v.loewis.de> <1233166249.6489.0.camel@adam-laptop> <4980A0C5.6000800@v.loewis.de> Message-ID: <1233169951.6489.1.camel@adam-laptop> Yep. Thats the plan, and, thanks again. On Wed, 2009-01-28 at 19:15 +0100, "Martin v. L?wis" wrote: > Adam Boduch wrote: > > Excellent. Thanks so much! > > You are welcome. I do recommend that you change the zipfile, though, to > remove one URL level. > > Regards, > Martin From martin at v.loewis.de Wed Jan 28 20:21:47 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 28 Jan 2009 20:21:47 +0100 Subject: [Catalog-sig] API Documentation In-Reply-To: <804e5c70901281055w64020000l78fbaa4086dbbb81@mail.gmail.com> References: <1233151272.6653.3.camel@adam-laptop> <4980996A.7030702@v.loewis.de> <1233166249.6489.0.camel@adam-laptop> <4980A0C5.6000800@v.loewis.de> <804e5c70901281055w64020000l78fbaa4086dbbb81@mail.gmail.com> Message-ID: <4980B04B.6050804@v.loewis.de> Lukasz Szybalski wrote: > How do you upload the documentation? Is there a link somewhere that > has instructions on getting these docs in. I might use it for my > package. Just go to your package's page, and it should be all obvious. Regards, Martin From szybalski at gmail.com Fri Jan 30 05:24:09 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 29 Jan 2009 22:24:09 -0600 Subject: [Catalog-sig] threads and xmlrpc? In-Reply-To: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com> References: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com> Message-ID: <804e5c70901292024u39782f74wbd5701f7b7ff1e81@mail.gmail.com> Hello, I'm running a threaded app using some calls via xmlrpc to pypi. What I'm trying to get is a to get a littler more responses in a shorter time, as I see that the bandwidth used by xmlrpc calls are minimal ( References: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com> <804e5c70901292024u39782f74wbd5701f7b7ff1e81@mail.gmail.com> Message-ID: <498293CE.4070606@v.loewis.de> > I'm running a threaded app using some calls via xmlrpc to pypi. What > I'm trying to get is a to get a littler more responses in a shorter > time, as I see that the bandwidth used by xmlrpc calls are minimal > ( after about 10min (~500 calls). I use a single connection and a queue > of 8 threads to get the data. Would anybody have an example on how to > run xmlrpc in a thread? Do I set multiple connections, or is there a > setting to keep the connection live or reconnect if disconnected? Using threads will not at all make it faster to communicate over a single connection. For a single connection, all communication must be serialized; you cannot issue a new request until the previous request has completed. So you might as well just issue the requests from a single thread. > Also, please advice if you think that somehow I am overloading your > servers. I've tasted some downloads speeds and I am sure you web > browser can accept 100+ requests per second, but what about xmlrpc? > Without threads I get <5 requests per second. I think 5 requests per second is fairly fast. Regards, Martin From szybalski at gmail.com Sat Jan 31 06:46:28 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Fri, 30 Jan 2009 23:46:28 -0600 Subject: [Catalog-sig] threads and xmlrpc? In-Reply-To: <498293CE.4070606@v.loewis.de> References: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com> <804e5c70901292024u39782f74wbd5701f7b7ff1e81@mail.gmail.com> <498293CE.4070606@v.loewis.de> Message-ID: <804e5c70901302146v43bb982bx222fd785bdff8af5@mail.gmail.com> On Thu, Jan 29, 2009 at 11:44 PM, "Martin v. L?wis" wrote: >> I'm running a threaded app using some calls via xmlrpc to pypi. What >> I'm trying to get is a to get a littler more responses in a shorter >> time, as I see that the bandwidth used by xmlrpc calls are minimal >> (> after about 10min (~500 calls). I use a single connection and a queue >> of 8 threads to get the data. Would anybody have an example on how to >> run xmlrpc in a thread? Do I set multiple connections, or is there a >> setting to keep the connection live or reconnect if disconnected? > > Using threads will not at all make it faster to communicate over a > single connection. For a single connection, all communication must > be serialized; you cannot issue a new request until the previous > request has completed. So you might as well just issue the requests > from a single thread. > >> Also, please advice if you think that somehow I am overloading your >> servers. I've tasted some downloads speeds and I am sure you web >> browser can accept 100+ requests per second, but what about xmlrpc? >> Without threads I get <5 requests per second. > > I think 5 requests per second is fairly fast. > Its more like 2 requests per second. If I set it to 2 threads I can list each package version in about an hour, but I lost connection when I was at a z packages. If I used 5-8 I can get half way in about 25min but I lose connection. ("Connection reset by peer") Would you know how can I issue more requests, and/or increase the number of connections? I know "http://www.faqs.org/rfcs/rfc2068.html See section 8.1.4. The RFC says "should limit 2 connections per server" and a lot of http client libraries obey this." Does xmlrpc lib used by pypi does the same? Does pypi use http://docs.python.org/library/xmlrpclib.html#multicall-objects This is my last try. I was hoping that I can increase the number of connections to at least 10/second ~20min but I can't seem to find any performance increases on xmlrpc. Is there another way to get: pypi.list_packages() pypi.package_releases('xyz') pypi.release_data(' xyz' ,' 0.7.79dev' ) If not then I guess I will go back to the regular for loop and loop through all the records in a serialized manner. (Its been 1h 15min and I am on packages starting with letter R.) Cpickle file coming soon for the metadata available in release_data for all packages. Thanks, Lucas