From rea at swing.tm.fr Wed Nov 1 18:54:15 2006 From: rea at swing.tm.fr (Joe Owens) Date: Wed, 1 Nov 2006 19:54:15 +0200 Subject: [Catalog-sig] martyrdom spontaneity Message-ID: <4548DF47.1010606@ffgc.org> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/catalog-sig/attachments/20061101/443cff55/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: lapse.gif Type: image/gif Size: 9227 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20061101/443cff55/attachment.gif From timcera at earthlink.net Mon Nov 6 02:47:30 2006 From: timcera at earthlink.net (Tim Cera) Date: Sun, 5 Nov 2006 20:47:30 -0500 Subject: [Catalog-sig] How to get best match URL to find source? Message-ID: <200611052047.32287.timcera@earthlink.net> I am trying to get the best match URL that points to source. How should I do that? I can't use easy_install since as near as I can figure, I can't force it to collect only source. I have tried something like below: pkgindex = PackageIndex() output = pkgindex.fetch_distribution(Requirement.parse(pkg), 'sandbox', force_scan=True, source=True) It does download the package, but I can't find the URL anywhere. Am I missing it? Is it possible to get to? Kindest regards, Tim Cera From pje at telecommunity.com Mon Nov 6 03:46:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Nov 2006 21:46:47 -0500 Subject: [Catalog-sig] How to get best match URL to find source? In-Reply-To: <200611052047.32287.timcera@earthlink.net> Message-ID: <5.1.1.6.0.20061105214154.01f21d68@sparrow.telecommunity.com> At 08:47 PM 11/5/2006 -0500, Tim Cera wrote: >I am trying to get the best match URL that points to source. How should >I do that? > >I can't use easy_install since as near as I can figure, I can't force it >to collect only source. See the --editable and --build-directory arguments. Simple usage: easy_install -eb. ProjectName Will download projectname's source (or check it out of Subversion), and unpack it to a directory called './projectname' (all lowercase). (The '.' is actually whatever directory you give to the -b or --build-directory option.) >I have tried something like below: > > pkgindex = PackageIndex() > output = pkgindex.fetch_distribution(Requirement.parse(pkg), 'sandbox', >force_scan=True, source=True) > >It does download the package, but I can't find the URL anywhere. Am I >missing it? Is it possible to get to? Not using fetch_distribution(), no. You would have to do the same steps that fetch_distribution() does to find it. From jim at zope.com Thu Nov 16 22:44:08 2006 From: jim at zope.com (Jim Fulton) Date: Thu, 16 Nov 2006 16:44:08 -0500 Subject: [Catalog-sig] PyPi down again Message-ID: <455CDBA8.5050806@zope.com> Any idea why now? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Thu Nov 16 22:48:28 2006 From: jim at zope.com (Jim Fulton) Date: Thu, 16 Nov 2006 16:48:28 -0500 Subject: [Catalog-sig] PyPi down again In-Reply-To: <455CDBA8.5050806@zope.com> References: <455CDBA8.5050806@zope.com> Message-ID: <455CDCAC.1000509@zope.com> Jim Fulton wrote: > Any idea why now? BTW, if someone will give me needed access and tell me how, I'd be happy to help out with recovery efforts, since I seem to notice lack of service most often. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From richardjones at optushome.com.au Thu Nov 16 22:59:24 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Fri, 17 Nov 2006 08:59:24 +1100 Subject: [Catalog-sig] PyPi down again In-Reply-To: <455CDBA8.5050806@zope.com> References: <455CDBA8.5050806@zope.com> Message-ID: <200611170859.24187.richardjones@optushome.com.au> On Friday 17 November 2006 08:44, Jim Fulton wrote: > Any idea why now? Just logged in: 23:00:48 up 30 days, 7:59, 2 users, load average: 34.41, 38.20, 39.90 There's a flood of apache processes, so I assume something's hitting the web server hard. I don't have access to the logs, so I can't say anythin beyond that. Richard From jim at zope.com Thu Nov 16 23:01:24 2006 From: jim at zope.com (Jim Fulton) Date: Thu, 16 Nov 2006 17:01:24 -0500 Subject: [Catalog-sig] PyPi down again In-Reply-To: <455CDBA8.5050806@zope.com> References: <455CDBA8.5050806@zope.com> Message-ID: <455CDFB4.7080701@zope.com> Jim Fulton wrote: > Any idea why now? FWIW, it seems to have been very slow/overloaded rather than down. It seems to recover for a bit and then become slow again. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Thu Nov 16 23:04:01 2006 From: jim at zope.com (Jim Fulton) Date: Thu, 16 Nov 2006 17:04:01 -0500 Subject: [Catalog-sig] PyPi down again In-Reply-To: <200611170859.24187.richardjones@optushome.com.au> References: <455CDBA8.5050806@zope.com> <200611170859.24187.richardjones@optushome.com.au> Message-ID: <455CE051.3050302@zope.com> Richard Jones wrote: > On Friday 17 November 2006 08:44, Jim Fulton wrote: >> Any idea why now? > > Just logged in: > > 23:00:48 up 30 days, 7:59, 2 users, load average: 34.41, 38.20, 39.90 > > There's a flood of apache processes, so I assume something's hitting the web > server hard. > > I don't have access to the logs, so I can't say anythin beyond that. Thanks for looking. Is there someone who can help? Can I help somehow? What can we do to protect ourselves from this in the future? (I still haven't looked at the developer info, so I'm not in a position to recommend anything, dang it.) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From richardjones at optushome.com.au Thu Nov 16 23:11:35 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Fri, 17 Nov 2006 09:11:35 +1100 Subject: [Catalog-sig] PyPi down again In-Reply-To: <455CE051.3050302@zope.com> References: <455CDBA8.5050806@zope.com> <200611170859.24187.richardjones@optushome.com.au> <455CE051.3050302@zope.com> Message-ID: <200611170911.35893.richardjones@optushome.com.au> On Friday 17 November 2006 09:04, Jim Fulton wrote: > Richard Jones wrote: > > On Friday 17 November 2006 08:44, Jim Fulton wrote: > >> Any idea why now? > > > > Just logged in: > > > > 23:00:48 up 30 days, 7:59, 2 users, load average: 34.41, 38.20, 39.90 > > > > There's a flood of apache processes, so I assume something's hitting the > > web server hard. > > > > I don't have access to the logs, so I can't say anythin beyond that. > > Thanks for looking. > > Is there someone who can help? Can I help somehow? We need someone with root access to look at the logs. I'm posting to pydotorg to see if we can get someone to look. Without seeing the logs I'm going to guess we've got a badly-behaved web spider hitting the browse interface. > What can we do to protect ourselves from this in the future? Assuming my guess about it being a spider hitting the browser interface, we already attempt to block spiders off from the browse interface both through the robots.txt file and by detecting their user-agent in the pypi code. Spiders don't *have* to read robots.txt, and my RE string for user-agent detection might be missing someone. We won't know until we can get someone to look at the logs. Richard From richardjones at optushome.com.au Thu Nov 16 23:30:29 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Fri, 17 Nov 2006 09:30:29 +1100 Subject: [Catalog-sig] PyPi down again In-Reply-To: <200611170911.35893.richardjones@optushome.com.au> References: <455CDBA8.5050806@zope.com> <455CE051.3050302@zope.com> <200611170911.35893.richardjones@optushome.com.au> Message-ID: <200611170930.29273.richardjones@optushome.com.au> On Friday 17 November 2006 09:11, Richard Jones wrote: > On Friday 17 November 2006 09:04, Jim Fulton wrote: > > Richard Jones wrote: > > > 23:00:48 up 30 days, 7:59, 2 users, load average: 34.41, 38.20, > > > 39.90 > > > > > > There's a flood of apache processes, so I assume something's hitting > > > the web server hard. > > > > > > I don't have access to the logs, so I can't say anythin beyond that. FWIW: 23:33:45 up 30 days, 8:32, 2 users, load average: 0.31, 1.67, 12.07 Richard From constant.beta at gmail.com Tue Nov 21 02:59:16 2006 From: constant.beta at gmail.com (=?ISO-8859-2?Q?Micha=B3_Kwiatkowski?=) Date: Tue, 21 Nov 2006 02:59:16 +0100 Subject: [Catalog-sig] PyPi down again In-Reply-To: <200611170930.29273.richardjones@optushome.com.au> References: <455CDBA8.5050806@zope.com> <455CE051.3050302@zope.com> <200611170911.35893.richardjones@optushome.com.au> <200611170930.29273.richardjones@optushome.com.au> Message-ID: <5e8b0f6b0611201759t4bdc6d3q233121cd28aa6318@mail.gmail.com> On 11/16/06, Richard Jones wrote: > > > > There's a flood of apache processes, so I assume something's hitting > > > > the web server hard. > > > > > > > > I don't have access to the logs, so I can't say anythin beyond that. > > FWIW: > > 23:33:45 up 30 days, 8:32, 2 users, load average: 0.31, 1.67, 12.07 Any news on the cause of last PyPI slowdown? Cheers, mk -- . o . >> http://joker.linuxstuff.pl << . . o It's easier to get forgiveness for being wrong o o o than forgiveness for being right. From richardjones at optushome.com.au Tue Nov 21 09:36:54 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Tue, 21 Nov 2006 19:36:54 +1100 Subject: [Catalog-sig] PyPi down again In-Reply-To: <5e8b0f6b0611201759t4bdc6d3q233121cd28aa6318@mail.gmail.com> References: <455CDBA8.5050806@zope.com> <200611170930.29273.richardjones@optushome.com.au> <5e8b0f6b0611201759t4bdc6d3q233121cd28aa6318@mail.gmail.com> Message-ID: <200611211936.54121.richardjones@optushome.com.au> On Tuesday 21 November 2006 12:59, Micha? Kwiatkowski wrote: > On 11/16/06, Richard Jones wrote: > > > > > There's a flood of apache processes, so I assume something's > > > > > hitting the web server hard. > > > > > > > > > > I don't have access to the logs, so I can't say anythin beyond > > > > > that. > > > > FWIW: > > > > 23:33:45 up 30 days, 8:32, 2 users, load average: 0.31, 1.67, 12.07 > > Any news on the cause of last PyPI slowdown? I've just put in a second request to pydotorg people. Richard