From monitor at jacobian.org Thu Sep 1 09:57:21 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Thu, 01 Sep 2011 02:57:21 -0500 Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection failed Message-ID: <1314863845.1@jacobian.org> Connection failed Service c.pypi.python.org Date: Thu, 01 Sep 2011 02:57:21 -0500 Action: alert Host: jacobian.org Description: failed, cannot open a connection to INET[c.pypi.python.org:80] via TCP Your faithful employee, monit From monitor at jacobian.org Thu Sep 1 10:14:42 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Thu, 01 Sep 2011 03:14:42 -0500 Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection succeeded Message-ID: <1314864884.1@jacobian.org> Connection succeeded Service c.pypi.python.org Date: Thu, 01 Sep 2011 03:14:41 -0500 Action: alert Host: jacobian.org Description: connection succeeded to INET[c.pypi.python.org:80] via TCP Your faithful employee, monit From richard at python.org Fri Sep 2 04:49:08 2011 From: richard at python.org (Richard Jones) Date: Fri, 2 Sep 2011 12:49:08 +1000 Subject: [Catalog-sig] Fwd: Cannot receive RSS feeds on latest updates to Packages In-Reply-To: References: Message-ID: >From Melton Low : > I am on a Mac running Safari. ?In the last few days, I get notifications of > new updates to the Packages on my link to the RSS URL?PyPI recent updates. > ?When I clink on my bookmarked link, no new items are downloaded. > The RSS feed on new additions to the Packages works fine. I can't see a problem. Please confirm the two urls you're referring to are: not working: http://pypi.python.org/pypi?%3Aaction=rss working: http://pypi.python.org/pypi?%3Aaction=packages_rss Looking at the RSS being generated it looks fine - plenty of recent content there. The last-modified and etag information appears to be correct as well, so caching is *probably* not an issue... Richard From monitor at jacobian.org Thu Sep 1 10:34:47 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Thu, 01 Sep 2011 03:34:47 -0500 Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection succeeded Message-ID: <1314866089.1@jacobian.org> Connection succeeded Service c.pypi.python.org Date: Thu, 01 Sep 2011 03:34:47 -0500 Action: alert Host: jacobian.org Description: connection succeeded to INET[c.pypi.python.org:80] via TCP Your faithful employee, monit From monitor at jacobian.org Thu Sep 1 10:26:16 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Thu, 01 Sep 2011 03:26:16 -0500 Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection failed Message-ID: <1314865579.1@jacobian.org> Connection failed Service c.pypi.python.org Date: Thu, 01 Sep 2011 03:26:16 -0500 Action: alert Host: jacobian.org Description: failed, cannot open a connection to INET[c.pypi.python.org:80] via TCP Your faithful employee, monit From monitor at jacobian.org Fri Sep 9 18:59:16 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Fri, 09 Sep 2011 11:59:16 -0500 Subject: [Catalog-sig] [monit] d.pypi.python.org - Connection succeeded Message-ID: <1315587557.1@jacobian.org> Connection succeeded Service d.pypi.python.org Date: Fri, 09 Sep 2011 11:59:16 -0500 Action: alert Host: jacobian.org Description: connection succeeded to INET[d.pypi.python.org:80] via TCP Your faithful employee, monit From monitor at jacobian.org Fri Sep 9 18:45:28 2011 From: monitor at jacobian.org (monitor at jacobian.org) Date: Fri, 09 Sep 2011 11:45:28 -0500 Subject: [Catalog-sig] [monit] d.pypi.python.org - Connection failed Message-ID: <1315586731.1@jacobian.org> Connection failed Service d.pypi.python.org Date: Fri, 09 Sep 2011 11:45:28 -0500 Action: alert Host: jacobian.org Description: failed, cannot open a connection to INET[d.pypi.python.org:80] via TCP Your faithful employee, monit From djay at pretaweb.com Mon Sep 12 10:43:11 2011 From: djay at pretaweb.com (Dylan Jay) Date: Mon, 12 Sep 2011 18:43:11 +1000 Subject: [Catalog-sig] parsing setuptools style requires.txt Message-ID: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Hi, A recent request[1] (which I didn't submit) was rejected with the following comment: "Assuming you are talking about setuptools-style dependencies: this has been rejected by the community, as hurting PEP 345. PEP-345-style dependencies are already parsed and provided. If you want to discuss further, post to catalog-sig." I had a search of the archives and couldn't locate the discussion, unless it's this one [2] which seemed to indicate that there was a suitable way to publish both pep345 compatible requirements as well as PIP and setuptools requirements via PYPI. It strikes me that 1. if someone is prepared to write a patch to pypi to handle setuptools style requirements 2. if there is a lot of packages out there and will continue to be for a long time, that have setuptools style dependency specification 3. if PEP345 isn't implemented in any tools yet (excuse my ignorance. I'm assuming PEP345 tool support is in distutils2 and that isn't finished yet) then why not allow a change to parsing of setuptools style dependencies? or perhaps point me to the discussion that explains what I'm missing. Its also been mentioned that in order to make this parsing work you need to run setup.py to get the requires.txt for setuptools packages and this is a security concern. However many packages already have the egg-info commend run before upload so there is no need to run setup.py. For those packages where there is a need I think security concerns could be overcome with the use of the restrictedpython package. Anything trying to import anything but the bare minimum is skipped. My interest in this is the idea that we could get distutils2 and/or zc.buildout to be able to download regular updates of metadata including dependencies, then perhaps those tools could avoid certain kinds of conflict errors which are a pain to debug without that information. For instance, the current design of zc.buildout means that: Download Bob. Bob 1.0 requires Fred >= 2.0. Download Fred 3.0 Download Marry. Marry 1.0 requires Fred < 2.5 Conflict error. Marry 1.0 requires Fred < 2.5 but we already have Fred 3.0. If instead we knew in advance of this conflict we could have chosen to download Fred 2.4 or at least warned the user there was a potential conflict and they should pick a compatible version. In the case of preinstalled packages, it could offer to downgrade Fred from 3.0 to 2.4. [1] http://sourceforge.net/tracker/?func=detail&aid=3407539&group_id=66150&atid=513503 [2] http://mail.python.org/pipermail/catalog-sig/2011-January/003452.html --- Dylan Jay Technical Solutions Manager PretaWeb: reducing duplication in the government web. P: +612 80819071 | M: +61421477460 | twitter.com/djay75 | linkedin.com/ in/djay75 From alexis at notmyidea.org Mon Sep 12 14:01:43 2011 From: alexis at notmyidea.org (=?ISO-8859-1?Q?Alexis_M=E9taireau?=) Date: Mon, 12 Sep 2011 14:01:43 +0200 Subject: [Catalog-sig] parsing setuptools style requires.txt In-Reply-To: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Message-ID: <4E6DF4A7.80900@notmyidea.org> Hi Dylan, On 09/12/2011 10:43 AM, Dylan Jay wrote: > ts also been mentioned that in order to make this parsing work you need > to run setup.py to get the requires.txt for setuptools packages and this > is a security concern. However many packages already have the egg-info > commend run before upload so there is no need to run setup.py. For those > packages where there is a need I think security concerns could be > overcome with the use of the restrictedpython package. Anything trying > to import anything but the bare minimum is skipped. One problem I can think about is the fact that it is not possible to get platform independent information coming from a setup.py, in the sense that setup.py is executed and can provide different metadata regarding the platform the setup.py is executing onto. PEP 345 and environment markers aims to resolve this problem. Bu, what are you trying to do exacty? Do you want to get the list of dependencies coming from setuptools? If yes, then the way we are doing it for the setuptools compatibility layer in distutils2/packaging is the following: 1. monkey-patch setuptools.setup with your own setup, storing the given metadata somewhere 2. run the setup.py which will call this monkey patched setup function This has been implemented here: http://bazaar.launchpad.net/~anybox/+junk/anybox.recipe.openerp/view/head:/anybox/recipe/openerp/__init__.py#L53 I agree this is not as easy as it could be with uploading metadata to pypi (as it will be with distutils2 and PEP 345) but having platform specific dependent metadata information on PyPI is only half answering the problem, so the solution is probably for now to do this on the client side. Hope this helps. -- Alexis From ziade.tarek at gmail.com Mon Sep 12 15:01:19 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 12 Sep 2011 15:01:19 +0200 Subject: [Catalog-sig] parsing setuptools style requires.txt In-Reply-To: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Message-ID: On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay wrote: ... > then why not allow a change to parsing of setuptools style dependencies? > or perhaps point me to the discussion that explains what I'm missing. > > Its also been mentioned that in order to make this parsing work you need to > run setup.py to get the requires.txt for setuptools packages and this is a > security concern. However many packages already have the egg-info commend > run before upload so there is no need to run setup.py. For those packages > where there is a need I think security concerns could be overcome with the > use of the restrictedpython package. Anything trying to import anything but > the bare minimum is skipped. the egg-info produced by setuptools is dependent on the platform it's run on, so running on the target client is what you need to get the right values for sure. with PEP 345-style metadata, you have ways to define static per-platform metadata. In fact, one of the goal is to have setup.cfg published at pypi > > My interest in this is the idea that we could get distutils2 and/or > zc.buildout to be able to download regular updates of metadata including > dependencies, then perhaps those tools could avoid certain kinds of conflict > errors which are a pain to debug without that information. For instance, the > current design of zc.buildout means that: > > Download Bob. Bob 1.0 requires Fred >= 2.0. > Download Fred 3.0 > Download Marry. Marry 1.0 requires Fred < 2.5 > Conflict error. Marry 1.0 requires Fred < 2.5 but we already have Fred 3.0. > > If instead we knew in advance of this conflict we could have chosen to > download Fred 2.4 or at least warned the user there was a potential conflict > and they should pick a compatible version. In the case of preinstalled > packages, it could offer to downgrade Fred from 3.0 to 2.4. I think Pip solves this by downloading all dependencies beforehand, no ? and it works for all platforms. The bottom line is that spending efforts on adding on server side platform-specific metadata that are not guaranteed to be accurate for your target platform, is not worth it since you can already do something accurate on client side at this time. PEP 345 will provide an improvement in the long term since you will be able to read dependencies, but also the file list, without having to download the package. IOW, installers will speed up because they will be able to build dependency graphs on the fly -- as long as all packages are PEP 345, which won't happen before a while. Cheers Tarek -- Tarek Ziad? | http://ziade.org From jim at zope.com Mon Sep 12 15:31:17 2011 From: jim at zope.com (Jim Fulton) Date: Mon, 12 Sep 2011 09:31:17 -0400 Subject: [Catalog-sig] parsing setuptools style requires.txt In-Reply-To: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Message-ID: On Mon, Sep 12, 2011 at 4:43 AM, Dylan Jay wrote: ... > My interest in this is the idea that we could get distutils2 and/or > zc.buildout to be able to download regular updates of metadata including > dependencies, then perhaps those tools could avoid certain kinds of conflict > errors which are a pain to debug without that information. For instance, the > current design of zc.buildout means that: > > Download Bob. Bob 1.0 requires Fred >= 2.0. > Download Fred 3.0 > Download Marry. Marry 1.0 requires Fred < 2.5 > Conflict error. Marry 1.0 requires Fred < 2.5 but we already have Fred 3.0. > If instead we knew in advance of this conflict we could have chosen to > download Fred 2.4 or at least warned the user there was a potential conflict > and they should pick a compatible version. In the case of preinstalled > packages, it could offer to downgrade Fred from 3.0 to 2.4. The issue here isn't that buildout has to download packages to get meta data. The problem is that it doesn't backtrack. This is compounded by fact that when computing and fetching requirements, it uses a depth-first strategy and doesn't use information about requirements of other packages in the working set until too late. There's definitely room for improvement in the strategy it uses, even short of backtracking. In any case, allowing buildout to get requirements info before downloading packages would be a useful optimization, but wouldn't solve the conflict problem. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From james.p.turk at gmail.com Tue Sep 13 00:37:48 2011 From: james.p.turk at gmail.com (James Turk) Date: Mon, 12 Sep 2011 18:37:48 -0400 Subject: [Catalog-sig] change package ownership on some packages Message-ID: Hi, I'm hoping that I can get ownership of a few packages that were uploaded to PyPI by my coworker that has departed and is currently out of contact (due to personal reasons, I can elaborate but would prefer not to on a public mailing list for his privacy). Due to the sudden nature of his departure this was one of the things overlooked. The packages in question are: The packages in question are: http://pypi.python.org/pypi/scrapelib http://pypi.python.org/pypi/jellyfish http://pypi.python.org/pypi/python-openstates http://pypi.python.org/pypi/nimsp http://pypi.python.org/pypi/name_tools You'll see that I'm listed (sometimes as jturk at sunlightfoundation.com) as co-author on most of these, they are also all hosted on the sunlightlabs github account (http://github.com/sunlightlabs), for which I'm one of the owners. I'd be happy to provide any additional verification that would be helpful to assert both of these facts. My pypi name is jamesturk, if that account could be added as a maintainer on these 5 I'd greatly appreciate it. Thanks, James -------------- next part -------------- An HTML attachment was scrubbed... URL: From djay at pretaweb.com Tue Sep 13 03:48:15 2011 From: djay at pretaweb.com (Dylan Jay) Date: Tue, 13 Sep 2011 11:48:15 +1000 Subject: [Catalog-sig] parsing setuptools style requires.txt In-Reply-To: References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Message-ID: On 12/09/2011, at 11:31 PM, Jim Fulton wrote: > On Mon, Sep 12, 2011 at 4:43 AM, Dylan Jay wrote: > ... >> My interest in this is the idea that we could get distutils2 and/or >> zc.buildout to be able to download regular updates of metadata >> including >> dependencies, then perhaps those tools could avoid certain kinds of >> conflict >> errors which are a pain to debug without that information. For >> instance, the >> current design of zc.buildout means that: >> >> Download Bob. Bob 1.0 requires Fred >= 2.0. >> Download Fred 3.0 >> Download Marry. Marry 1.0 requires Fred < 2.5 >> Conflict error. Marry 1.0 requires Fred < 2.5 but we already have >> Fred 3.0. > >> If instead we knew in advance of this conflict we could have chosen >> to >> download Fred 2.4 or at least warned the user there was a potential >> conflict >> and they should pick a compatible version. In the case of >> preinstalled >> packages, it could offer to downgrade Fred from 3.0 to 2.4. > > The issue here isn't that buildout has to download packages > to get meta data. The problem is that it doesn't backtrack. This is > compounded by fact that when computing and fetching > requirements, it uses a depth-first strategy and doesn't use > information about requirements of other packages in the working > set until too late. There's definitely room for improvement in the > strategy it uses, even short of backtracking. true but to do backtracking properly could result in downloading many different available versions of potentially large packages which isn't so nice. > > In any case, allowing buildout to get requirements info before > downloading packages would be a useful optimization, but wouldn't > solve the conflict problem. yes, backtracking would :) or at least it would solve some of them. Do you think it's doable even if there's no pypi api to get requirements? I think this would be a great feature as conflict errors are very frustrating for new users. They also tend come up in buildout more than any other tool due to the way to it's normally used and new users tend to blame buildout for this. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton From djay at pretaweb.com Tue Sep 13 03:53:37 2011 From: djay at pretaweb.com (Dylan Jay) Date: Tue, 13 Sep 2011 11:53:37 +1000 Subject: [Catalog-sig] parsing setuptools style requires.txt In-Reply-To: References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Message-ID: On 12/09/2011, at 11:01 PM, Tarek Ziad? wrote: > On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay wrote: > ... >> then why not allow a change to parsing of setuptools style >> dependencies? >> or perhaps point me to the discussion that explains what I'm missing. >> >> Its also been mentioned that in order to make this parsing work you >> need to >> run setup.py to get the requires.txt for setuptools packages and >> this is a >> security concern. However many packages already have the egg-info >> commend >> run before upload so there is no need to run setup.py. For those >> packages >> where there is a need I think security concerns could be overcome >> with the >> use of the restrictedpython package. Anything trying to import >> anything but >> the bare minimum is skipped. > > the egg-info produced by setuptools is dependent on the platform it's > run on, so running on the target client is what you need to get the > right values for sure. true. but I'd guess that 99% of eggs have the same set of requirements regardless of platform. And if we had someway to label these requirements on pypi as "dodgy- setuptools" and should be used only as a hint rather the exact set of requirements for your particular platform? > > with PEP 345-style metadata, you have ways to define static > per-platform metadata. > > In fact, one of the goal is to have setup.cfg published at pypi > > >> >> My interest in this is the idea that we could get distutils2 and/or >> zc.buildout to be able to download regular updates of metadata >> including >> dependencies, then perhaps those tools could avoid certain kinds of >> conflict >> errors which are a pain to debug without that information. For >> instance, the >> current design of zc.buildout means that: >> >> Download Bob. Bob 1.0 requires Fred >= 2.0. >> Download Fred 3.0 >> Download Marry. Marry 1.0 requires Fred < 2.5 >> Conflict error. Marry 1.0 requires Fred < 2.5 but we already have >> Fred 3.0. >> >> If instead we knew in advance of this conflict we could have chosen >> to >> download Fred 2.4 or at least warned the user there was a potential >> conflict >> and they should pick a compatible version. In the case of >> preinstalled >> packages, it could offer to downgrade Fred from 3.0 to 2.4. > > I think Pip solves this by downloading all dependencies beforehand, no > ? and it works for all platforms. yes it can be solved by downloading all dependencies but you'd have to download all versions of all dependencies potentially to avoid a conflict in the worst case. > > The bottom line is that spending efforts on adding on server side > platform-specific metadata that are not guaranteed to be accurate for > your target platform, is not worth it since you can already do > something accurate on client side at this time. see above. > > PEP 345 will provide an improvement in the long term since you will be > able to read dependencies, but also the file list, without having to > download the package. > > IOW, installers will speed up because they will be able to build > dependency graphs on the fly -- as long as all packages are PEP 345, > which won't happen before a while. can't wait, but in the meantime I can't see why installers can't use these hints to speed things up as long as they use it with the caveat that requirements could potentially differ once actually installed. > > Cheers > Tarek > > -- > Tarek Ziad? | http://ziade.org From jim at zope.com Tue Sep 13 15:33:08 2011 From: jim at zope.com (Jim Fulton) Date: Tue, 13 Sep 2011 09:33:08 -0400 Subject: [Catalog-sig] parsing setuptools style requires.txt In-Reply-To: References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Message-ID: On Mon, Sep 12, 2011 at 9:48 PM, Dylan Jay wrote: > On 12/09/2011, at 11:31 PM, Jim Fulton wrote: > >> On Mon, Sep 12, 2011 at 4:43 AM, Dylan Jay wrote: ... >> In any case, allowing buildout to get requirements info before >> downloading packages would be a useful optimization, but wouldn't >> solve the conflict problem. > > yes, backtracking would :) or at least it would solve some of them. Do you > think it's doable even if there's no pypi api to get requirements? I'm sure backtracking is doable. I suspect it's rather hard. In any case, it's orthogonal to avoiding downloading to get requirements. > I think this would be a great feature as conflict errors are very > frustrating for new users. They also tend come up in buildout more than any > other tool due to the way to it's normally used and new users tend to blame > buildout for this. Maybe. In the interest of being incremental, I think it would be better start with improving the current algorithm short of backtracking. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From ziade.tarek at gmail.com Tue Sep 13 17:24:06 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 13 Sep 2011 17:24:06 +0200 Subject: [Catalog-sig] parsing setuptools style requires.txt In-Reply-To: References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com> Message-ID: On Tue, Sep 13, 2011 at 3:53 AM, Dylan Jay wrote: > On 12/09/2011, at 11:01 PM, Tarek Ziad? wrote: > >> On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay wrote: >> ... >>> >>> then why not allow a change to parsing of setuptools style dependencies? >>> or perhaps point me to the discussion that explains what I'm missing. >>> >>> Its also been mentioned that in order to make this parsing work you need >>> to >>> run setup.py to get the requires.txt for setuptools packages and this is >>> a >>> security concern. However many packages already have the egg-info commend >>> run before upload so there is no need to run setup.py. For those packages >>> where there is a need I think security concerns could be overcome with >>> the >>> use of the restrictedpython package. Anything trying to import anything >>> but >>> the bare minimum is skipped. >> >> the egg-info produced by setuptools is dependent on the platform it's >> run on, so running on the target client is what you need to get the >> right values for sure. > > true. but I'd guess that 99% of eggs have the same set of requirements > regardless of platform. where did you get this percentage from ? I see this pattern more often than in 1% of the cases: if sys.platform == 'win32': reqs.append('pywin32') (I have more examples) > And if we had someway to label these requirements on pypi as > "dodgy-setuptools" and should be used only as a hint rather the exact set of > requirements for your particular platform? how would you detect them ? ... >> I think Pip solves this by downloading all dependencies beforehand, no >> ? and it works for all platforms. > > yes it can be solved by downloading all dependencies but you'd have to > download all versions of all dependencies potentially to avoid a conflict in > the worst case. > >> >> The bottom line is that spending efforts on adding on server side >> platform-specific metadata that are not guaranteed to be accurate for >> your target platform, is not worth it since you can already do >> something accurate on client side at this time. > > see above. > > >> >> PEP 345 will provide an improvement in the long term since you will be >> able to read dependencies, but also the file list, without having to >> download the package. >> >> IOW, installers will speed up because they will be able to build >> dependency graphs on the fly -- as long as all packages are PEP 345, >> which won't happen before a while. > > can't wait, but in the meantime I can't see why installers can't use these > hints to speed things up as long as they use it with the caveat that > requirements could potentially differ once actually installed. a speed up with a potential non-working installation, does not sound like a good idea me. now, installers could keep a cache of platform-specific metadata on the client side to build a dependency graph that's looked up and completed on any new installation. That would save some calls and downloads. Cheers Tarek -- Tarek Ziad? | http://ziade.org From techtonik at gmail.com Wed Sep 14 14:12:35 2011 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 14 Sep 2011 15:12:35 +0300 Subject: [Catalog-sig] PyPI in Mercurial, conversion cleanup and state of Selenium tests Message-ID: Hi, Please, CC. I've converted PyPI repository to Mercurial to enable public posting of changes to BitBucket or Google Code. You can find it at https://bitbucket.org/techtonik/pypi If people like to use Mercurial for PyPI development, it may worth to switch completely. For that purpose it is worthy to cleanup the clone. I've noticed a pypi_selenium/ dir dated 2005, which contains whole JUnit with Xerces binary.What is the status of this stuff? http://wiki.python.org/moin/CheeseShopDev is silent about any testing at all. It is also interesting to know about status of these branches: egginfo 1675:b5ea39a3f498 tarek-pep-345-support 1455:cbf5f43e3917 tarek-pypi 1114:1db82f5e78b2 name-mangling 767:e6d118808d09 redesign 694:88eb8de4246c MichaelTwomey 466:c56de66b3da2 richard 4:84a72b2a8497 pypi 1672:fa3073ebc9a9 (inactive) -- anatoly t. From drkjam at gmail.com Wed Sep 14 14:36:57 2011 From: drkjam at gmail.com (drkjam) Date: Wed, 14 Sep 2011 13:36:57 +0100 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations References: Message-ID: Would it be possible to have trove classifiers added to PyPI specifically for PyPy and possibly also Jython and IronPython? Regards, David Moss > From martin at v.loewis.de Wed Sep 14 15:22:13 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Sep 2011 15:22:13 +0200 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: References: Message-ID: <4E70AA85.6090700@v.loewis.de> Am 14.09.11 14:36, schrieb drkjam: > Would it be possible to have trove classifiers added to PyPI specifically for PyPy and possibly also Jython and IronPython? Here's the standard set of questions for proposers of new classifiers: * what specific classifiers are you proposing? * what specific semantics would you imply for packages tagged with these classifiers? * what specific packages that are already on PyPI would be tagged with these specific classifiers? Regards, Martin From fuzzyman at gmail.com Thu Sep 15 13:55:42 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 15 Sep 2011 12:55:42 +0100 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: <4E70AA85.6090700@v.loewis.de> References: <4E70AA85.6090700@v.loewis.de> Message-ID: On 14 September 2011 14:22, "Martin v. L?wis" wrote: > Am 14.09.11 14:36, schrieb drkjam: > > Would it be possible to have trove classifiers added to PyPI specifically >> for PyPy and possibly also Jython and IronPython? >> > > Here's the standard set of questions for proposers of new classifiers: > > * what specific classifiers are you proposing? > * what specific semantics would you imply for packages tagged with these > classifiers? > * what specific packages that are already on PyPI would be tagged with > these specific classifiers? > Addressing the last 2 of these questions particularly: It would be useful to be able to specify that a package is tested with (known to work with) alternative implementations. So, for example, I would tag "mock" with pypy and Jython as I routinely test on those implementations and know that it works. As a user (or potential user) of pypy, it would be very useful to know what packages are known work with pypy. At the moment pypy has to maintain its own list or users have to check individual package documentation or proceed by trial and error. For Jython and IronPython it is possible to have packages that *only* work on those implementations because they integrate with or use features of the underlying platform. For example rst2xaml is a package only useful for those using IronPython (either on the desktop or with Silverlight). All the best, Michael > > Regards, > Martin > > ______________________________**_________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/**mailman/listinfo/catalog-sig > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at gmail.com Thu Sep 15 14:09:08 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 15 Sep 2011 13:09:08 +0100 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: <4E70AA85.6090700@v.loewis.de> References: <4E70AA85.6090700@v.loewis.de> Message-ID: On 14 September 2011 14:22, "Martin v. L?wis" wrote: > Am 14.09.11 14:36, schrieb drkjam: > > Would it be possible to have trove classifiers added to PyPI specifically >> for PyPy and possibly also Jython and IronPython? >> > > Here's the standard set of questions for proposers of new classifiers: > > * what specific classifiers are you proposing? > Suggestion: Programming Language - Python - Implementation - CPython Programming Language - Python - Implementation - pypy Programming Language - Python - Implementation - jython Programming Language - Python - Implementation - IronPython Michael Foord > * what specific semantics would you imply for packages tagged with these > classifiers? > * what specific packages that are already on PyPI would be tagged with > these specific classifiers? > > Regards, > Martin > > ______________________________**_________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/**mailman/listinfo/catalog-sig > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Sep 15 16:12:02 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Sep 2011 16:12:02 +0200 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: References: <4E70AA85.6090700@v.loewis.de> Message-ID: <4E7207B2.4070404@v.loewis.de> > Programming Language - Python - Implementation - CPython > Programming Language - Python - Implementation - pypy > Programming Language - Python - Implementation - jython > Programming Language - Python - Implementation - IronPython Opinions on this proposal? (including the specific spelling, leaving alone that the separator is ::, not -) Regards, Martin From mal at egenix.com Thu Sep 15 16:24:05 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 15 Sep 2011 16:24:05 +0200 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: <4E7207B2.4070404@v.loewis.de> References: <4E70AA85.6090700@v.loewis.de> <4E7207B2.4070404@v.loewis.de> Message-ID: <4E720A85.2070501@egenix.com> "Martin v. L?wis" wrote: > >> Programming Language - Python - Implementation - CPython >> Programming Language - Python - Implementation - pypy >> Programming Language - Python - Implementation - jython >> Programming Language - Python - Implementation - IronPython > > Opinions on this proposal? (including the specific spelling, > leaving alone that the separator is ::, not -) Better user CPython, PyPy and Jython for consistency with the other Trove spellings. I'm -0 on the "Implementation" part. Do we really need this ? Also: What about release versions of those implementations ? Jython and IronPython appear to follow the CPython release versions, but PyPy uses its own version scheme. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 15 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 19 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From fuzzyman at gmail.com Thu Sep 15 16:30:46 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 15 Sep 2011 15:30:46 +0100 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: <4E720A85.2070501@egenix.com> References: <4E70AA85.6090700@v.loewis.de> <4E7207B2.4070404@v.loewis.de> <4E720A85.2070501@egenix.com> Message-ID: On 15 September 2011 15:24, M.-A. Lemburg wrote: > "Martin v. L?wis" wrote: > > > >> Programming Language - Python - Implementation - CPython > >> Programming Language - Python - Implementation - pypy > >> Programming Language - Python - Implementation - jython > >> Programming Language - Python - Implementation - IronPython > > > > Opinions on this proposal? (including the specific spelling, > > leaving alone that the separator is ::, not -) > > Better user CPython, PyPy and Jython for consistency with the > other Trove spellings. > > I'm -0 on the "Implementation" part. Do we really need this ? > Jython, PyPy and CPython are not "programming languages" they're implementations of the Python programming language - so I would prefer to differentiate like this. > > Also: What about release versions of those implementations ? > > We could always add those later if anyone really wanted them (for Python we have the broad "Python" *plus* version specific categories). In other words, YAGNI (for now). Michael > Jython and IronPython appear to follow the CPython release > versions, but PyPy uses its own version scheme. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Sep 15 2011) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2011-10-04: PyCon DE 2011, Leipzig, Germany 19 days to go > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Thu Sep 15 16:57:54 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 15 Sep 2011 16:57:54 +0200 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: Message from =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= of "Thu, 15 Sep 2011 16:12:02 +0200." <4E7207B2.4070404@v.loewis.de> References: <4E70AA85.6090700@v.loewis.de> <4E7207B2.4070404@v.loewis.de> Message-ID: <201109151457.p8FEvs6M021638@theraft.openend.se> In a message of Thu, 15 Sep 2011 16:12:02 +0200, "Martin v. L?wis" writes: > >> Programming Language - Python - Implementation - CPython >> Programming Language - Python - Implementation - pypy >> Programming Language - Python - Implementation - jython >> Programming Language - Python - Implementation - IronPython > >Opinions on this proposal? (including the specific spelling, >leaving alone that the separator is ::, not -) > >Regards, >Martin PyPy should be spelled like that. Laura From exarkun at twistedmatrix.com Thu Sep 15 16:36:10 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 15 Sep 2011 14:36:10 -0000 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: <4E720A85.2070501@egenix.com> References: <4E70AA85.6090700@v.loewis.de> <4E7207B2.4070404@v.loewis.de> <4E720A85.2070501@egenix.com> Message-ID: <20110915143610.1808.320688955.divmod.xquotient.870@localhost.localdomain> On 02:24 pm, mal at egenix.com wrote: >"Martin v. L?wis" wrote: >> >>>Programming Language - Python - Implementation - CPython >>>Programming Language - Python - Implementation - pypy >>>Programming Language - Python - Implementation - jython >>>Programming Language - Python - Implementation - IronPython >> >>Opinions on this proposal? (including the specific spelling, >>leaving alone that the separator is ::, not -) > >Better user CPython, PyPy and Jython for consistency with the >other Trove spellings. > >I'm -0 on the "Implementation" part. Do we really need this ? These seem like a different sort of thing than the existing classifiers like Programming Language :: Python :: 2.7 It seems that each implementation is trying to provide compatibility with a particular CPython release. You could imagine a project specifying Programming Language :: Python :: 2.7 Programming Language :: Python :: Implementation :: Jython To indicate that a Jython release implementing Python 2.7 is required, and only the Jython runtime is supported. >Also: What about release versions of those implementations ? > >Jython and IronPython appear to follow the CPython release >versions, but PyPy uses its own version scheme. The current release of PyPy is 1.6, but it is intended to be compatible with CPython 2.7. If the classifiers are interpreted how I've interpreted them above, then combining two or more is sufficient, and having PyPy :: 1.5, PyPy 1.6, PyPy :: 1.7 doesn't offer much. However, as usual, interpretation is up in the air, since a classifier is only a name, with no associated documented meaning, beyond what people manage to glean from mailing lists or by looking at existing projects that have decided to use them. Jean-Paul From mal at egenix.com Fri Sep 16 01:18:34 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 16 Sep 2011 01:18:34 +0200 Subject: [Catalog-sig] PyPI trove classifiers for alternate language implementations In-Reply-To: References: <4E70AA85.6090700@v.loewis.de> <4E7207B2.4070404@v.loewis.de> <4E720A85.2070501@egenix.com> Message-ID: <4E7287CA.6070900@egenix.com> Michael Foord wrote: > On 15 September 2011 15:24, M.-A. Lemburg wrote: > >> "Martin v. L?wis" wrote: >>> >>>> Programming Language - Python - Implementation - CPython >>>> Programming Language - Python - Implementation - pypy >>>> Programming Language - Python - Implementation - jython >>>> Programming Language - Python - Implementation - IronPython >>> >>> Opinions on this proposal? (including the specific spelling, >>> leaving alone that the separator is ::, not -) >> >> Better user CPython, PyPy and Jython for consistency with the >> other Trove spellings. >> >> I'm -0 on the "Implementation" part. Do we really need this ? >> > > > Jython, PyPy and CPython are not "programming languages" they're > implementations of the Python programming language - so I would prefer to > differentiate like this. Well, yes, but if you look at the OS section of the Trove list you also find different implementations of BSD under the BSD category: Operating System :: POSIX :: BSD Operating System :: POSIX :: BSD :: BSD/OS Operating System :: POSIX :: BSD :: FreeBSD Operating System :: POSIX :: BSD :: NetBSD Operating System :: POSIX :: BSD :: OpenBSD (just to take one exmaple) See http://pypi.python.org/pypi?%3Aaction=list_classifiers for the full list. Also note that Cython is listed as: Programming Language :: Cython even Zope is listed as programming language: Programming Language :: Zope so there isn't all that much consistency in the naming. BTW: Stackless is missing from your list. >> Also: What about release versions of those implementations ? >> >> > We could always add those later if anyone really wanted them (for Python we > have the broad "Python" *plus* version specific categories). In other words, > YAGNI (for now). > > Michael > > >> Jython and IronPython appear to follow the CPython release >> versions, but PyPy uses its own version scheme. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 16 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Tue Sep 27 11:40:44 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Sep 2011 11:40:44 +0200 Subject: [Catalog-sig] Fwd: The state of PyPI In-Reply-To: References: Message-ID: I have sent that to the PSF list because there's a PSF project about PyPI infra. But someone complained, saying that I was doing this discussion "behind closed doors" SInce this is not my goal, I am now spamming more lists... ---------- Forwarded message ---------- From: Tarek Ziad? Date: Tue, Sep 27, 2011 at 10:37 AM Subject: The state of PyPI To: PSF Members List Cc: Richard Jones , Steve Holden Hey This is just a mail that summarizes the current state of PyPI, the existing features, and what can be done next to improve stuff. I am sending this in the PSF members list because we had a project of an infrastructure going on, and I want to make sure all involved parties are in the same page. 1/ stability and high availability 2/ private mirrors 3/ private projects 4/ tutorial ? = stability and high availability = we went in two directions to improve PyPI : 1/ add the mirroring protocol 2/ make the PyPI server more reliable by pushing its storage in a redundant cloud. == mirroring == The mirroring protocol (PEP 381) is implemented on server-side, I've worked with Martin on this, and we have mirrors now: Look at http://pypi.python.org/mirrors Also, there's a client that anyone can use to set up a mirror: http://pypi.python.org/pypi/pep381client The idea is that anyone in the community willing to maintain a mirror can do so. We add the mirror in the CND, and make it available for client tools to use. What's really missing right now is more integration on client-side. - Pip supports the mirroring protocol, and can fall back to a mirror, but I am not 100% sure this is a default behavior. ?(please correct me if it is now) - Buildout knows how to use *another server* than the main PyPI, so can manually switch to a mirror, but I don't think it's transparent. It should. - Distribute/Setuptools does not do anything for this, and should. - everything is already implemented in packaging/distutils2 The effect of the mirrors is that PyPI being down should not impact the community. This will be true once all tool are transparently using the mirrors. == better infra == I think the project is staled right now. = private mirrors = Having a private mirror makes a lot of sense, when companies need to make sure their build systems are not relying on external services like PyPI or a mirror. It's also a good way to dramatically reduce the load for the community servers. The idea is that a Jenkins server that builds hundreds of Python apps every hour should not hammer PyPI. We have everything needed these days to set up this kind of system, with zc.buildout or pip good practices. What we need is a good tutorial or a guide [*] = private projects = The part that we do not address in the community is private projects: since we don't have any permissions/group/roles system in PyPI, everything is public. One way to solve this is to have a local repository for private packages, that is looked by tools like pip or easy_install, with the --find-links option. What we need is a good tutorial or a guide [*] = tutorial = [*] If this helps, I am willing to work on a tutorial day for Pycon US, that goes through all of this, to help people set up their dev. environment the best way possible. The material could then be published at python.org/pypi to help out. I know Richard has some material already, so maybe this could be a joint tutorial ? HTH Cheers Tarek -- Tarek Ziad? | http://ziade.org -- Tarek Ziad? | http://ziade.org From mal at egenix.com Tue Sep 27 12:04:25 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 27 Sep 2011 12:04:25 +0200 Subject: [Catalog-sig] Fwd: The state of PyPI In-Reply-To: References: Message-ID: <4E819FA9.7020300@egenix.com> Tarek Ziad? wrote: > = stability and high availability = > > we went in two directions to improve PyPI : > > 1/ add the mirroring protocol > 2/ make the PyPI server more reliable by pushing its storage in a > redundant cloud. > ... > == better infra == > > I think the project is staled right now. True, I don't have the needed cycles to make progress and neither do the team members. We do have the Amazon infrastructure setup (EC2, CloudFront, S3), and some progress has been made to sync the S3 storage with the local copies of the packages. The next steps would be to implement a trigger based sync scheme for packages and the simple/ index, i.e. new uploads should trigger an S3 copy process and update the simple/ index on S3 as well. For more info, please have a look at the wiki page: http://wiki.python.org/moin/CloudPyPI Here's the original proposal: http://wiki.python.org/moin/CloudPyPI/Proposal -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 27 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-10-04: PyCon DE 2011, Leipzig, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From carl at oddbird.net Tue Sep 27 16:36:19 2011 From: carl at oddbird.net (Carl Meyer) Date: Tue, 27 Sep 2011 08:36:19 -0600 Subject: [Catalog-sig] Fwd: The state of PyPI In-Reply-To: References: Message-ID: <4E81DF63.9080703@oddbird.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Tarek, On 09/27/2011 03:40 AM, Tarek Ziad? wrote: > = private projects = > > The part that we do not address in the community is private projects: > since we don't have any permissions/group/roles system in PyPI, > everything is public. > > One way to solve this is to have a local repository for private > packages, that is looked by tools like pip or easy_install, with the > --find-links option. > > > What we need is a good tutorial or a guide [*] > > = tutorial = > > [*] If this helps, I am willing to work on a tutorial day for Pycon > US, that goes through all of this, to help people set up their dev. > environment the best way possible. > > The material could then be published at python.org/pypi to help out. > > I know Richard has some material already, so maybe this could be a > joint tutorial ? The talk I gave at djangocon.us a couple weeks ago [1] covers this material, though certainly not as in-depth as it could be covered, and with a focus primarily on pip. I would be willing/interested in doing (or co-presenting) a PyCon tutorial on solutions for managing local package repositories. Carl [1] https://carljm.github.com/tamingdeps/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6B32MACgkQ8W4rlRKtE2dOdACeNnd7egS3z3agCkvJMAXIjNbS 90sAoLrd8X50uA7n1Vz6uBcRtXd5Fpyc =RRy1 -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Tue Sep 27 17:17:25 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Sep 2011 17:17:25 +0200 Subject: [Catalog-sig] Fwd: The state of PyPI In-Reply-To: <4E81DF63.9080703@oddbird.net> References: <4E81DF63.9080703@oddbird.net> Message-ID: On Tue, Sep 27, 2011 at 4:36 PM, Carl Meyer wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Tarek, > > On 09/27/2011 03:40 AM, Tarek Ziad? wrote: >> = private projects = >> >> The part that we do not address in the community is private projects: >> since we don't have any permissions/group/roles system in PyPI, >> everything is public. >> >> One way to solve this is to have a local repository for private >> packages, that is looked by tools like pip or easy_install, with the >> --find-links option. >> >> >> What we need is a good tutorial or a guide [*] >> >> = tutorial = >> >> [*] If this helps, I am willing to work on a tutorial day for Pycon >> US, that goes through all of this, to help people set up their dev. >> environment the best way possible. >> >> The material could then be published at python.org/pypi to help out. >> >> I know Richard has some material already, so maybe this could be a >> joint tutorial ? > > The talk I gave at djangocon.us a couple weeks ago [1] covers this > material, though certainly not as in-depth as it could be covered, and > with a focus primarily on pip. > > I would be willing/interested in doing (or co-presenting) a PyCon > tutorial on solutions for managing local package repositories. > > Carl > > ?[1] https://carljm.github.com/tamingdeps/ Awesome. Richard and Alexis are also interested, so I propose that we work on a joint proposal on an etherpad and submit it. I do believe half a day would be enough > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk6B32MACgkQ8W4rlRKtE2dOdACeNnd7egS3z3agCkvJMAXIjNbS > 90sAoLrd8X50uA7n1Vz6uBcRtXd5Fpyc > =RRy1 > -----END PGP SIGNATURE----- > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Tarek Ziad? | http://ziade.org From noah at coderanger.net Tue Sep 27 19:24:02 2011 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 27 Sep 2011 10:24:02 -0700 Subject: [Catalog-sig] Fwd: The state of PyPI In-Reply-To: References: Message-ID: I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final. On Sep 27, 2011, at 2:40 AM, Tarek Ziad? wrote: > > == better infra == > The current plan is to have two primary load balancer VMs running Nginx acting as both balancer and SSL termination points. These will share a set of floating IPs using Heartbeat. Behind this will be the same Apache configuration currently in use (Apache serving static files and PyPI running as an FCGI script controlled by Apache) running on two VMs, both talking to the same master-slave Postgres 9 replication setup. Package files will initially be handled by a shared DRBD drive, however this may be obsoleted by the project to move file hosting to Cloudfront or another CDN. A currently open question is how best to provide reliability and security for the SSH-based file upload system currently deployed on ximenez. Most likely we can setup the initial SSH endpoint on the load balancers to run a proxy to one of the main PyPI application servers, however failover would have to be semi-manual (possibly driven by Chef, meaning a chef-client run would have to happen before the tunnel would be updated, or anywhere up to 30 minutes). Given the relatively minimal public knowledge of this service, I think this is acceptable as a first-pass but a future solution involving HAProxy or another TCP load balancer to handle the SSH traffic might be appropriate. Similarly, automated database failover is not planned at this time due to the extra application-side complexity however the process will be well documented and able to be executed by all tier one on-call operations staff if the Postgres master goes offline for some reason. This will only partially address the current reliability issues as many of the current problems are linked to Apache or mod_fcgid needing to be restarted. In that light I would like to see PyPI and the catalog-sig group investigate moving the codebase to work against mod_wsgi or gunicorn (no real preference between the two) to create a more reliable runtime environment. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ziade.tarek at gmail.com Tue Sep 27 19:33:58 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 27 Sep 2011 19:33:58 +0200 Subject: [Catalog-sig] Fwd: The state of PyPI In-Reply-To: References: Message-ID: On Tue, Sep 27, 2011 at 7:24 PM, Noah Kantrowitz wrote: > I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final. Thanks, I've started this page with all the content : http://wiki.python.org/moin/BetterPyPI -- Tarek Ziad? | http://ziade.org From noah at coderanger.net Tue Sep 27 19:34:55 2011 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 27 Sep 2011 10:34:55 -0700 Subject: [Catalog-sig] Fwd: The state of PyPI In-Reply-To: References: Message-ID: <5F826E99-A368-47E2-A26D-B691F8A05649@coderanger.net> On Sep 27, 2011, at 10:33 AM, Tarek Ziad? wrote: > On Tue, Sep 27, 2011 at 7:24 PM, Noah Kantrowitz wrote: >> I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final. > > Thanks, > > I've started this page with all the content : > http://wiki.python.org/moin/BetterPyPI If people have feedback please also make sure to let me know so I we can discuss it. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: