From jim at zope.com Mon Oct 16 19:48:44 2006 From: jim at zope.com (Jim Fulton) Date: Mon, 16 Oct 2006 13:48:44 -0400 Subject: [Catalog-sig] PyPI overloaded(?) Message-ID: <4533C5FC.6050506@zope.com> PyPI seems to e down or severly overloaded. Does anyone have any idea why? Is it time to set up some mirrors? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Tue Oct 17 15:13:04 2006 From: jim at zope.com (Jim Fulton) Date: Tue, 17 Oct 2006 09:13:04 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <4533C5FC.6050506@zope.com> References: <4533C5FC.6050506@zope.com> Message-ID: <4534D6E0.1080109@zope.com> Jim Fulton wrote: > PyPI seems to e down or severly overloaded. Does anyone have any > idea why? Is it time to set up some mirrors? I'm a bit disappointed that there was no written response to this. Shortly after I sent this, the behavior of pypi improved quite a bit. I don't know if this was in response to my post or if it just got better on it's own. As we adopt setuptools and zc.buildout, pypi is becoming a critical resource that needs to be available. I need to know if I can rely on www.python.org/pypi, or if I need to look at doing something else. My preference, by far, is to rely on pypi. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Tue Oct 17 15:14:42 2006 From: jim at zope.com (Jim Fulton) Date: Tue, 17 Oct 2006 09:14:42 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <4534D6E0.1080109@zope.com> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> Message-ID: <4534D742.3060603@zope.com> Jim Fulton wrote: > Jim Fulton wrote: >> PyPI seems to e down or severly overloaded. Does anyone have any >> idea why? Is it time to set up some mirrors? > > I'm a bit disappointed that there was no written response to this. > Shortly after I sent this, the behavior of pypi improved > quite a bit. I don't know if this was in response to my post > or if it just got better on it's own. > > As we adopt setuptools and zc.buildout, pypi is becoming a critical > resource that needs to be available. I need to know if I can > rely on www.python.org/pypi, or if I need to look at doing something > else. My preference, by far, is to rely on pypi. I should have noted in my note that my intent is not to complain. Mainly, I want to make sure that we we're in agreement about the goals and to make sure the PSF allocates the rsources needed to achieve them. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Tue Oct 17 19:37:16 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 17 Oct 2006 19:37:16 +0200 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <4534D742.3060603@zope.com> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com> Message-ID: <453514CC.8000206@v.loewis.de> Jim Fulton schrieb: > I should have noted in my note that my intent is not to complain. > Mainly, I want to make sure that we we're in agreement about the goals > and to make sure the PSF allocates the rsources needed to achieve them. Can you volunteer yourself or somebody else to help in maintaining PyPI? Regards, Martin From martin at v.loewis.de Tue Oct 17 20:08:35 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 17 Oct 2006 20:08:35 +0200 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de> <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> Message-ID: <45351C23.4090003@v.loewis.de> Ben Bangert schrieb: > I'll volunteer to help out in maintenance. I'm in agreement with Jim > that PyPI definitely needs some sort of mirror system. I'm not so sure this is the definite answer. If the system is overloaded, it might be because it consumes too many resources itself (in which case mirroring wouldn't help), or because something else on the machine is consuming too many resources (in which case the installation should be moved elsewhere entirely). Mirroring it is quite a challenge, as you have to consider write operations to the mirror. I think efforts should first go into improving the performance. > While I currently > advocate that people using setuptools based installations maintain their > own egg repo in case PyPI is unavailable, it'd be great to have mirrors. > On a side-note, I don't know who maintains PyPI now, but thank you for > the great job, for a single un-mirrored site, the reliability thus far > has been great. It's the primary author, Richard Jones. To my knowledge, he does both source code improvements, as well as maintaining the installation. You might want to start with creating an installation on a test machine and study its performance. Regards, Martin From jim at zope.com Tue Oct 17 20:21:09 2006 From: jim at zope.com (Jim Fulton) Date: Tue, 17 Oct 2006 14:21:09 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <453514CC.8000206@v.loewis.de> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de> Message-ID: <45351F15.6090707@zope.com> Martin v. L?wis wrote: > Jim Fulton schrieb: >> I should have noted in my note that my intent is not to complain. >> Mainly, I want to make sure that we we're in agreement about the goals >> and to make sure the PSF allocates the rsources needed to achieve them. > > Can you volunteer yourself or somebody else to help in maintaining > PyPI? Possibly. Is it an issue of people? I really don't know what the cause of the problems yesterday were. Was it a software problem? Load? I'd like to understand what the issues are first. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From ben at groovie.org Tue Oct 17 19:50:51 2006 From: ben at groovie.org (Ben Bangert) Date: Tue, 17 Oct 2006 10:50:51 -0700 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <453514CC.8000206@v.loewis.de> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de> Message-ID: <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> On Oct 17, 2006, at 10:37 AM, Martin v. L?wis wrote: > Jim Fulton schrieb: >> I should have noted in my note that my intent is not to complain. >> Mainly, I want to make sure that we we're in agreement about the >> goals >> and to make sure the PSF allocates the rsources needed to achieve >> them. > > Can you volunteer yourself or somebody else to help in maintaining > PyPI? I'll volunteer to help out in maintenance. I'm in agreement with Jim that PyPI definitely needs some sort of mirror system. While I currently advocate that people using setuptools based installations maintain their own egg repo in case PyPI is unavailable, it'd be great to have mirrors. On a side-note, I don't know who maintains PyPI now, but thank you for the great job, for a single un-mirrored site, the reliability thus far has been great. Cheers, Ben From jim at zope.com Tue Oct 17 20:26:54 2006 From: jim at zope.com (Jim Fulton) Date: Tue, 17 Oct 2006 14:26:54 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de> <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> Message-ID: <4535206E.10008@zope.com> Ben Bangert wrote: > On Oct 17, 2006, at 10:37 AM, Martin v. L?wis wrote: > >> Jim Fulton schrieb: >>> I should have noted in my note that my intent is not to complain. >>> Mainly, I want to make sure that we we're in agreement about the goals >>> and to make sure the PSF allocates the rsources needed to achieve them. >> >> Can you volunteer yourself or somebody else to help in maintaining >> PyPI? > > I'll volunteer to help out in maintenance. I'm in agreement with Jim > that PyPI definitely needs some sort of mirror system. While I currently > advocate that people using setuptools based installations maintain their > own egg repo in case PyPI is unavailable, I'm not sure that is very practical. We're using lots of eggs that we don't produce. They may depend on still more that we don't produce... > it'd be great to have mirrors. The only obvious way to have a usable separate repo is to create a mirror and I'm afraid having a lot of (or a few) people creating mirrors could end up crushing pypi in the process. I think this will require some coordination. > On a side-note, I don't know who maintains PyPI now, but thank you for > the great job, Absolutely! > for a single un-mirrored site, the reliability thus far > has been great. Well, yesterday wasn't the first day I had a problem. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Tue Oct 17 20:36:21 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 17 Oct 2006 20:36:21 +0200 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <45351F15.6090707@zope.com> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de> <45351F15.6090707@zope.com> Message-ID: <453522A5.7040203@v.loewis.de> Jim Fulton schrieb: >>> I should have noted in my note that my intent is not to complain. >>> Mainly, I want to make sure that we we're in agreement about the goals >>> and to make sure the PSF allocates the rsources needed to achieve them. >> Can you volunteer yourself or somebody else to help in maintaining >> PyPI? > > Possibly. Is it an issue of people? I really don't know what the > cause of the problems yesterday were. Was it a software problem? > Load? I'd like to understand what the issues are first. Primarily, it's people. Very few people understand the PyPI code well, and nearly nobody has time research performance problems. My guess is that the code could be improved to perform certain actions better. For example, it's know that PyPI cannot really cope with web crawlers too well. If too many of them hit him at a bad spot, it will go down. It takes man-power to investigate these problems and propose solutions; solutions based on speculation (like "let's mirror") are likely wrong. I'm not sure whether there is any way to investigate performance problems after the fact; if there is, it likely isn't possible in this case. So one would have to define a performance analysis procedure first, implement it, and wait for the performance to go down again to understand what's commonly causing the problem. Of course, if somebody would log in and analyse *while* the problem is happening, there might be a much better chance at understanding the problem. OTOH, it's really no fun to investigate something remotely on an overloaded machine, so likely, whoever is in charge would just make the problem go away instead of studying it. Regards, Martin From amk at amk.ca Tue Oct 17 21:11:23 2006 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 17 Oct 2006 15:11:23 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <45351F15.6090707@zope.com> References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de> <45351F15.6090707@zope.com> Message-ID: <20061017191123.GA8987@rogue.amk.ca> On Tue, Oct 17, 2006 at 02:21:09PM -0400, Jim Fulton wrote: > Possibly. Is it an issue of people? I really don't know what the > cause of the problems yesterday were. Was it a software problem? > Load? I'd like to understand what the issues are first. Unclear. When your catalog-sig e-mail first arrived yesterday, I logged in to ximinez to see if the system was overloaded, and uptime reported load averages something like 0.2, 0.5, 0.7, which made it look like the system wasn't very loaded, but was coming down from a greater load. (A rude crawler, maybe?) I didn't investigate further, because it didn't look like there was any problem at the time that needed fixing. --amk From richardjones at optushome.com.au Wed Oct 18 06:43:49 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Wed, 18 Oct 2006 14:43:49 +1000 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <45351C23.4090003@v.loewis.de> References: <4533C5FC.6050506@zope.com> <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> <45351C23.4090003@v.loewis.de> Message-ID: <200610181443.50005.richardjones@optushome.com.au> Sorry I didn't respond in a more immediate manner - I'm quite busy with work and organising papers for OSDC '06. On Wednesday 18 October 2006 04:08, Martin v. L?wis wrote: > I'm not so sure this is the definite answer. If the system is > overloaded, it might be because it consumes too many resources itself > (in which case mirroring wouldn't help), or because something else > on the machine is consuming too many resources (in which case the > installation should be moved elsewhere entirely). We still have the problem that the PyPI browse interface is quite CPU-intensive and if it's hit by a bot it'll definitely impact on overall system performance. We have a check in the browse code to see if the user agent matches: botre = re.compile(r'^$|brains|yeti|myie2|findlinks|ia_archiver|psycheclone| badass|crawler|slurp|spider|bot|scooter|infoseek|looksmart|jeeves', re.I) and if it does then the browse returns an empty page. This RE is pretty complete - I use it to redirect bots to a dedicated ZEO client at work. I've added a robots.txt to http://cheeseshop.python.org (I always meant to, but never got around to it). Unfortunately, I'm not sure "Disallow: /pypi?:action=browse" will be handled properly. We'll see. Richard From richardjones at optushome.com.au Wed Oct 18 06:45:23 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Wed, 18 Oct 2006 14:45:23 +1000 Subject: [Catalog-sig] Cheese shop: search In-Reply-To: <4517FAEA.3010801@colorstudy.com> References: <4517FAEA.3010801@colorstudy.com> Message-ID: <200610181445.23914.richardjones@optushome.com.au> On Tuesday 26 September 2006 01:51, Ian Bicking wrote: > Can we please get the search back? If someone writes it, sure. The old search was far less useful that the Google search is now. Richard From jim at zope.com Wed Oct 18 13:20:05 2006 From: jim at zope.com (Jim Fulton) Date: Wed, 18 Oct 2006 07:20:05 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <200610181443.50005.richardjones@optushome.com.au> References: <4533C5FC.6050506@zope.com> <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> <45351C23.4090003@v.loewis.de> <200610181443.50005.richardjones@optushome.com.au> Message-ID: <45360DE5.10007@zope.com> Richard Jones wrote: > Sorry I didn't respond in a more immediate manner - I'm quite busy with work > and organising papers for OSDC '06. > > On Wednesday 18 October 2006 04:08, Martin v. L?wis wrote: >> I'm not so sure this is the definite answer. If the system is >> overloaded, it might be because it consumes too many resources itself >> (in which case mirroring wouldn't help), or because something else >> on the machine is consuming too many resources (in which case the >> installation should be moved elsewhere entirely). > > We still have the problem that the PyPI browse interface is quite > CPU-intensive and if it's hit by a bot it'll definitely impact on overall > system performance. Is that what happened the other day? > We have a check in the browse code to see if the user agent matches: > > botre = re.compile(r'^$|brains|yeti|myie2|findlinks|ia_archiver|psycheclone| > badass|crawler|slurp|spider|bot|scooter|infoseek|looksmart|jeeves', re.I) > > and if it does then the browse returns an empty page. This RE is pretty > complete - I use it to redirect bots to a dedicated ZEO client at work. > > I've added a robots.txt to http://cheeseshop.python.org (I always meant to, > but never got around to it). Unfortunately, I'm not > sure "Disallow: /pypi?:action=browse" will be handled properly. We'll see. Is there something somewhere that documents the architecture? I'd like to try to offer helpful suggestions, but without more knowledge of what's going on, that's hard to do. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From richardjones at optushome.com.au Wed Oct 18 13:46:54 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Wed, 18 Oct 2006 21:46:54 +1000 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <45360DE5.10007@zope.com> References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com> Message-ID: <200610182146.54648.richardjones@optushome.com.au> On Wednesday 18 October 2006 21:20, Jim Fulton wrote: > Richard Jones wrote: > > We still have the problem that the PyPI browse interface is quite > > CPU-intensive and if it's hit by a bot it'll definitely impact on overall > > system performance. > > Is that what happened the other day? Sorry, I'm just guessing. I don't have the time to look into it in more detail at the moment. > Is there something somewhere that documents the architecture? No, at the moment there's just the code. Richard From jim at zope.com Wed Oct 18 15:39:58 2006 From: jim at zope.com (Jim Fulton) Date: Wed, 18 Oct 2006 09:39:58 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <200610182146.54648.richardjones@optushome.com.au> References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com> <200610182146.54648.richardjones@optushome.com.au> Message-ID: <45362EAE.6090308@zope.com> Richard Jones wrote: > On Wednesday 18 October 2006 21:20, Jim Fulton wrote: >> Richard Jones wrote: >>> We still have the problem that the PyPI browse interface is quite >>> CPU-intensive and if it's hit by a bot it'll definitely impact on overall >>> system performance. >> Is that what happened the other day? > > Sorry, I'm just guessing. I don't have the time to look into it in more detail > at the moment. Perhaps someone should look at the logs. Ben Bangert volunteered to help out. :) I can help out some too, if necessary. >> Is there something somewhere that documents the architecture? > > No, at the moment there's just the code. Is the code in a publicly available repository? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From amk at amk.ca Wed Oct 18 15:58:19 2006 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 18 Oct 2006 09:58:19 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <45362EAE.6090308@zope.com> References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com> <200610182146.54648.richardjones@optushome.com.au> <45362EAE.6090308@zope.com> Message-ID: <20061018135819.GA18165@localhost.localdomain> On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote: > Perhaps someone should look at the logs. Ben Bangert > volunteered to help out. :) I can help out some too, if necessary. Does Bangert have SSH access to python.org? If not, I suggest he write to pydotorg at python.org and provide an SSH key. I can add him to ximinez. (And is he someone you know reasonably well? Is he a Zope Corp. employee?) > Is the code in a publicly available repository? https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one point it used SQLite, but I don't know if Richard left traces of SQLite support in the code. --amk From jim at zope.com Wed Oct 18 16:04:50 2006 From: jim at zope.com (Jim Fulton) Date: Wed, 18 Oct 2006 10:04:50 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <20061018135819.GA18165@localhost.localdomain> References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com> <200610182146.54648.richardjones@optushome.com.au> <45362EAE.6090308@zope.com> <20061018135819.GA18165@localhost.localdomain> Message-ID: <45363482.7050108@zope.com> A.M. Kuchling wrote: > On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote: >> Perhaps someone should look at the logs. Ben Bangert >> volunteered to help out. :) I can help out some too, if necessary. > > Does Bangert have SSH access to python.org? If not, I suggest he > write to pydotorg at python.org and provide an SSH key. I can add him to > ximinez. > > (And is he someone you know reasonably well? Is he a Zope > Corp. employee?) I don't know him at all, other than through this list. He is not a ZC employee. >> Is the code in a publicly available repository? > > https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one > point it used SQLite, but I don't know if Richard left traces of > SQLite support in the code. Thanks. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From richardjones at optushome.com.au Wed Oct 18 23:21:36 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Thu, 19 Oct 2006 07:21:36 +1000 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <20061018135819.GA18165@localhost.localdomain> References: <4533C5FC.6050506@zope.com> <45362EAE.6090308@zope.com> <20061018135819.GA18165@localhost.localdomain> Message-ID: <200610190721.37079.richardjones@optushome.com.au> On Wednesday 18 October 2006 23:58, A.M. Kuchling wrote: > On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote: > > Is the code in a publicly available repository? > > https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one > point it used SQLite, but I don't know if Richard left traces of > SQLite support in the code. Follow the "Developers" link from the http://cheeseshop.python.org/pypi sidebar for more information. Richard From jim at zope.com Wed Oct 18 23:49:35 2006 From: jim at zope.com (Jim Fulton) Date: Wed, 18 Oct 2006 17:49:35 -0400 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <200610190721.37079.richardjones@optushome.com.au> References: <4533C5FC.6050506@zope.com> <45362EAE.6090308@zope.com> <20061018135819.GA18165@localhost.localdomain> <200610190721.37079.richardjones@optushome.com.au> Message-ID: <4536A16F.1010204@zope.com> Richard Jones wrote: > On Wednesday 18 October 2006 23:58, A.M. Kuchling wrote: >> On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote: >>> Is the code in a publicly available repository? >> https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one >> point it used SQLite, but I don't know if Richard left traces of >> SQLite support in the code. > > Follow the "Developers" link from the http://cheeseshop.python.org/pypi > sidebar for more information. Thanks. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From ianb at colorstudy.com Thu Oct 19 00:14:59 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 18 Oct 2006 17:14:59 -0500 Subject: [Catalog-sig] PyPI overloaded(?) In-Reply-To: <20061018135819.GA18165@localhost.localdomain> References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com> <200610182146.54648.richardjones@optushome.com.au> <45362EAE.6090308@zope.com> <20061018135819.GA18165@localhost.localdomain> Message-ID: <4536A763.5020904@colorstudy.com> A.M. Kuchling wrote: > On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote: >> Perhaps someone should look at the logs. Ben Bangert >> volunteered to help out. :) I can help out some too, if necessary. > > Does Bangert have SSH access to python.org? If not, I suggest he > write to pydotorg at python.org and provide an SSH key. I can add him to > ximinez. > > (And is he someone you know reasonably well? Is he a Zope > Corp. employee?) I can vouch for Ben; he's the developer of Pylons (http://pylonshq.com/) -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From ianb at colorstudy.com Fri Oct 20 18:15:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 20 Oct 2006 11:15:46 -0500 Subject: [Catalog-sig] Cheese shop: search In-Reply-To: <200610181445.23914.richardjones@optushome.com.au> References: <4517FAEA.3010801@colorstudy.com> <200610181445.23914.richardjones@optushome.com.au> Message-ID: <4538F632.4010302@colorstudy.com> Richard Jones wrote: > On Tuesday 26 September 2006 01:51, Ian Bicking wrote: >> Can we please get the search back? > > If someone writes it, sure. The old search was far less useful that the Google > search is now. I thought my email outlined why the Google search is completely useless. To summarize: it's useless. The old search was a little awkward, but it worked. Simply putting it back as it was would be a step forward. And actually the only improvement I'd like to see in the old search is a way to search all fields at once, and then a search form on the front page -- but even without these the search is functional. Well, if we wanted to be really fancy we'd want a full text indexer in the database, which probably isn't installed, and then create a field that contained the full text of each entry by simply concatenating items, and probably giving weight to some metadata by doing something scrappy like: full_text = title + title + keywords + keywords + topics + topics + description Otherwise if we are sticking with Google, cheese shop at least needs to tell google not to index the browse pages. And probably needs to give google a sitemap so it will index all the packages more prompty, as it's not indexing nearly often enough right now. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From fdrake at gmail.com Fri Oct 20 18:41:38 2006 From: fdrake at gmail.com (Fred Drake) Date: Fri, 20 Oct 2006 12:41:38 -0400 Subject: [Catalog-sig] Cheese shop: search In-Reply-To: <4538F632.4010302@colorstudy.com> References: <4517FAEA.3010801@colorstudy.com> <200610181445.23914.richardjones@optushome.com.au> <4538F632.4010302@colorstudy.com> Message-ID: <9cee7ab80610200941l72a06d38tc81be933293777de@mail.gmail.com> On 10/20/06, Ian Bicking wrote: > I thought my email outlined why the Google search is completely useless. > To summarize: it's useless. The old search was a little awkward, but > it worked. Simply putting it back as it was would be a step forward. Agreed! -Fred -- Fred L. Drake, Jr. "Every sin is the result of a collaboration." --Lucius Annaeus Seneca From timcera at earthlink.net Thu Oct 26 06:51:36 2006 From: timcera at earthlink.net (Tim Cera) Date: Thu, 26 Oct 2006 00:51:36 -0400 Subject: [Catalog-sig] Add requirements to XMLRPC? Message-ID: <200610260051.36324.timcera@earthlink.net> Hello, It would really be nice for a project that I am working on to get the package requirements from XMLRPC. What are the issues with adding the requirements to XMLPRC? Kindest regards, Tim Cera From richardjones at optushome.com.au Thu Oct 26 23:43:38 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Fri, 27 Oct 2006 07:43:38 +1000 Subject: [Catalog-sig] Add requirements to XMLRPC? In-Reply-To: <200610260051.36324.timcera@earthlink.net> References: <200610260051.36324.timcera@earthlink.net> Message-ID: <200610270743.38255.richardjones@optushome.com.au> On Thursday 26 October 2006 14:51, Tim Cera wrote: > It would really be nice for a project that I am working on to get the > package requirements from XMLRPC. What are the issues with adding the > requirements to XMLPRC? Someone has to write the code. I check it, commit it and install it. http://wiki.python.org/moin/CheeseShopDev has more info. Richard From jeremy.kloth at 4suite.org Sat Oct 28 20:12:53 2006 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Sat, 28 Oct 2006 12:12:53 -0600 Subject: [Catalog-sig] Add requirements to XMLRPC? In-Reply-To: <200610270743.38255.richardjones@optushome.com.au> References: <200610260051.36324.timcera@earthlink.net> <200610270743.38255.richardjones@optushome.com.au> Message-ID: <200610281212.54391.jeremy.kloth@4suite.org> On Thursday 26 October 2006 3:43 pm, Richard Jones wrote: > On Thursday 26 October 2006 14:51, Tim Cera wrote: > > It would really be nice for a project that I am working on to get the > > package requirements from XMLRPC. What are the issues with adding the > > requirements to XMLPRC? > > Someone has to write the code. I check it, commit it and install it. The attached patch adds the 'requires', 'provides' and 'obsoletes' fields to the release_data() RPC method. Note this is untested, but copied from webui.py where is builds the PKG-INFO. -- Jeremy Kloth http://4suite.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: rpc.py Type: application/x-python Size: 3063 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20061028/e41249b7/attachment.bin From jeremy.kloth at 4suite.org Sat Oct 28 20:22:40 2006 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Sat, 28 Oct 2006 12:22:40 -0600 Subject: [Catalog-sig] Add requirements to XMLRPC? In-Reply-To: <200610281212.54391.jeremy.kloth@4suite.org> References: <200610260051.36324.timcera@earthlink.net> <200610270743.38255.richardjones@optushome.com.au> <200610281212.54391.jeremy.kloth@4suite.org> Message-ID: <200610281222.40239.jeremy.kloth@4suite.org> On Saturday 28 October 2006 12:12 pm, Jeremy Kloth wrote: > On Thursday 26 October 2006 3:43 pm, Richard Jones wrote: > > On Thursday 26 October 2006 14:51, Tim Cera wrote: > > > It would really be nice for a project that I am working on to get the > > > package requirements from XMLRPC. What are the issues with adding the > > > requirements to XMLPRC? > > > > Someone has to write the code. I check it, commit it and install it. > > The attached patch adds the 'requires', 'provides' and 'obsoletes' fields > to the release_data() RPC method. > > Note this is untested, but copied from webui.py where is builds the > PKG-INFO. Maybe this time I'll attach the patch, not the whole file. :) -- Jeremy Kloth http://4suite.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: rpc.patch Type: text/x-diff Size: 622 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20061028/356f270a/attachment.bin From dangoor at gmail.com Sun Oct 29 19:21:21 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Sun, 29 Oct 2006 13:21:21 -0500 Subject: [Catalog-sig] cheeseshop down? Message-ID: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> It looks like the cheeseshop is down. "Bad Gateway" error from the python.org link and I can't get a connection at all on cheeseshop.python.org Kevin From jeremy.kloth at 4suite.org Mon Oct 30 07:00:08 2006 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Sun, 29 Oct 2006 23:00:08 -0700 Subject: [Catalog-sig] Add requirements to XMLRPC? In-Reply-To: <200610281222.40239.jeremy.kloth@4suite.org> References: <200610260051.36324.timcera@earthlink.net> <200610281212.54391.jeremy.kloth@4suite.org> <200610281222.40239.jeremy.kloth@4suite.org> Message-ID: <200610292300.08918.jeremy.kloth@4suite.org> On Saturday 28 October 2006 12:22 pm, Jeremy Kloth wrote: > > On Thursday 26 October 2006 3:43 pm, Richard Jones wrote: > > > Someone has to write the code. I check it, commit it and install it. This patch adds the information available for the 'files' action to the RPC release_urls() command. -- Jeremy Kloth http://4suite.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: rpc.patch Type: text/x-diff Size: 782 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20061029/a0c7ae45/attachment.bin From richardjones at optushome.com.au Mon Oct 30 08:21:31 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Mon, 30 Oct 2006 18:21:31 +1100 Subject: [Catalog-sig] cheeseshop down? In-Reply-To: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> Message-ID: <200610301821.31281.richardjones@optushome.com.au> On Monday 30 October 2006 05:21, Kevin Dangoor wrote: > It looks like the cheeseshop is down. "Bad Gateway" error from the > python.org link and I can't get a connection at all on > cheeseshop.python.org Appeared to be a temporary thing (it's back). I don't have time to look into what might have happened right now though. Richard From jim at zope.com Mon Oct 30 12:08:23 2006 From: jim at zope.com (Jim Fulton) Date: Mon, 30 Oct 2006 06:08:23 -0500 Subject: [Catalog-sig] cheeseshop down? In-Reply-To: <200610301821.31281.richardjones@optushome.com.au> References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au> Message-ID: <4545DD27.9060502@zope.com> Richard Jones wrote: > On Monday 30 October 2006 05:21, Kevin Dangoor wrote: >> It looks like the cheeseshop is down. "Bad Gateway" error from the >> python.org link and I can't get a connection at all on >> cheeseshop.python.org > > Appeared to be a temporary thing (it's back). Did it recover on it's own? Or did someone do something to bring it back? > I don't have time to look into > what might have happened right now though. This is a fairly serious problem. In the past, you questioned why anyone would want to implement their own package indexes. A major reason is reliability. Personally, I'd prefer that there be only one package index, but it has to be very available, at least the parts that setuptools uses. PyPI is great, but it needs to be much more available. I'll refrain, for now, from making suggestions, although I really want to, because I haven't had a chance to study the current implementation. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From amk at amk.ca Mon Oct 30 13:59:19 2006 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 30 Oct 2006 07:59:19 -0500 Subject: [Catalog-sig] cheeseshop down? In-Reply-To: <4545DD27.9060502@zope.com> References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au> <4545DD27.9060502@zope.com> Message-ID: <20061030125919.GA4868@localhost.localdomain> On Mon, Oct 30, 2006 at 06:08:23AM -0500, Jim Fulton wrote: > Did it recover on it's own? Or did someone do something > to bring it back? Someone edited the web server's config file and broke it; Anthony Baxter fixed it later. --amk From jim at zope.com Mon Oct 30 15:54:38 2006 From: jim at zope.com (Jim Fulton) Date: Mon, 30 Oct 2006 09:54:38 -0500 Subject: [Catalog-sig] cheeseshop down? In-Reply-To: <20061030125919.GA4868@localhost.localdomain> References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au> <4545DD27.9060502@zope.com> <20061030125919.GA4868@localhost.localdomain> Message-ID: <4546122E.2080408@zope.com> A.M. Kuchling wrote: > On Mon, Oct 30, 2006 at 06:08:23AM -0500, Jim Fulton wrote: >> Did it recover on it's own? Or did someone do something >> to bring it back? > > Someone edited the web server's config file and broke it; Anthony > Baxter fixed it later. Ah, cool. Thanks. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Mon Oct 30 22:08:20 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Oct 2006 22:08:20 +0100 Subject: [Catalog-sig] cheeseshop down? In-Reply-To: <4545DD27.9060502@zope.com> References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au> <4545DD27.9060502@zope.com> Message-ID: <454669C4.6080206@v.loewis.de> Jim Fulton schrieb: >> I don't have time to look into what might have happened right now >> though. > > This is a fairly serious problem. In the past, you questioned why > anyone would want to implement their own package indexes. And I would still question that. I can understand why people are unhappy if the service isn't reliable; I can't understand why they then think they can do better their own. > I'll refrain, for now, from making suggestions, although I really > want to, because I haven't had a chance to study the current > implementation. I hope the suggestion won't be "let's build another package database". Other suggestions certainly would be welcome. As we found in this specific case, it really was an admin error, by some unknown admin. Would your suggestion have such error from happening? Regards, Martin From jim at zope.com Mon Oct 30 22:29:42 2006 From: jim at zope.com (Jim Fulton) Date: Mon, 30 Oct 2006 16:29:42 -0500 Subject: [Catalog-sig] cheeseshop down? In-Reply-To: <454669C4.6080206@v.loewis.de> References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au> <4545DD27.9060502@zope.com> <454669C4.6080206@v.loewis.de> Message-ID: <45466EC6.2020606@zope.com> Martin v. L?wis wrote: > Jim Fulton schrieb: >>> I don't have time to look into what might have happened right now >>> though. >> This is a fairly serious problem. In the past, you questioned why >> anyone would want to implement their own package indexes. > > And I would still question that. I can understand why people are > unhappy if the service isn't reliable; I can't understand why they > then think they can do better their own. Because the current one is down or unresponsive more often then many of us will be able to deal with. >> I'll refrain, for now, from making suggestions, although I really >> want to, because I haven't had a chance to study the current >> implementation. > > I hope the suggestion won't be "let's build another package database". That wasn't what I had in mind, no. > Other suggestions certainly would be welcome. As we found in this > specific case, it really was an admin error, by some unknown admin. I guess the fact that the admin who caused the problem is unknown is a problem in itself. > Would your suggestion have such error from happening? I don't know. It might have, depending on what it was. :) I wouldn't make a suggestion without finding out more about the implementation. It has been suggested in the past that we might want to create mirrors. That *would* have helped in this situation. Please understand that I'm not trying to criticize anyone. PyPI is a great system, when it works. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org