[TriZPUG] Plone server very sluggish-update

Wed Aug 17 17:49:39 CEST 2011

I think Paul was asking if your code is in development mode on the
production server.  Dev mode or debug mode causes a lot of extra stuff
to happen to track what's going wrong, and is never the right thing to
do in production.

I don't use plone, so I'm not sure how it's set, but make sure your code
is in production mode.

On Wed, 2011-08-17 at 15:24 +0000, Kraus, Robert E wrote:
> Thanks for all the suggestions that have been posted. Some new ideas to look into.
> Yes, this is the production server. Our dev server simply does not get enough traffic to see whether changes have made any difference.
> And no, no custom code that I am aware of at least.
> 
> What we done/have had done in the last day or two:
>   Blocked robots by user agent via the squid caching proxy (robots that are known to be sloppy, don’t follow the rules, or otherwise are not important)
>   Added a 3rd zope instance
>   Made sure that all security patches have been applied to all instances
>   Increased zodb_cache to 30,000 from default
> 
> Ran the new buildout late yesterday afternoon, and after stopping and restarting all zope instances, we noticed a pretty major improvement. However, we will see when the load is really going to get heavy tomorrow.
> 
> Still had a problem with major sluggishness pretty consistently between 6:00-6:30pm on a daily basis, and yesterday was no exception. Am going to dig deeper into the logs to see if I can figure out what is going on at this time.
> 
> Should we be compacting the zope db and restarting our zope instances on a daily basis. We sometimes notice a short lived improvement after doing this, but no reason to do more work than is actually beneficial. Also, we will begin a more regular powering off of our Plone VM.
> 
> Still open to more suggestions, if you have any. Thanks again.
> 
> Robert Kraus, Web Services
> FPG Child Development Institute
> University of North Carolina at Chapel Hill
> Campus Box 8185
> Chapel Hill, NC 27599-8040
> 
> robert_kraus at unc.edu<mailto:robert_kraus at unc.edu>
> phone: 919.962.7310
> 
> From: trizpug-bounces+robert_kraus=unc.edu at python.org [mailto:trizpug-bounces+robert_kraus=unc.edu at python.org] On Behalf Of Paul Godley
> Sent: Wednesday, August 17, 2011 10:56 AM
> To: Triangle (North Carolina) Zope and Python Users Group
> Subject: Re: [TriZPUG] Urgent Help needed-Plone server very sluggish
> 
> Confirm you're in production mode and not development mode?
> 
> On Aug 16, 2011, at 9:32 AM, "Kraus, Robert E" <robert_kraus at unc.edu<mailto:robert_kraus at unc.edu>> wrote:
> Plone gurus,
> My supervisor, John Garcia, suggested that I reach out to the local Plone community for some help with a very perplexing problem which has us stumped.
> [My email address changed a while back and consequently I do not have access to the Trizpug list. Feel free to post this if you wish.]
> 
> Server: Windows 2003 Plone 3.x server, with Apache in front with a Squid caching proxy in front of Plone. Two Zope instances are started, one is for anonymous users and the other for authenticated users (heavy usage by authenticated users was making the site sluggish for anonymous users). The Squid cache proxy (and the Plone buildout for the most part) was set up by SixFeetUp since we were unfamiliar with the Squid configuration (with regards to Plone, at least).
> 
> Our problem is this: since June, this Plone instance, which is only running one site, has continued to get slower in its page request response times. Recently it has virtually ground to a halt at times. Plone reboots, zopedb compaction, even server reboots help some but only temporarily. The site’s manager, Jonathan, reported this the past several evenings evenings:
> 
> 
>   *   We started experiencing sluggishness in June.
>   *   Erratic extreme sluggishness has been going on for several weeks.
>   *   Most days recently, on arrival page edits times have been about 1-2 minute plus (typically 5-10 seconds on a good day).
>   *   Several persons have reported page load times were very slow throughout the day.
>   *   Server stopped responding completely about 10 minutes before this initial email.
> 
> This group/site has some heavy use with associates across the country starting in the next day or so and we are very concerned that the site will not be fully functional for them. There are occasional days, or periods within a day, in which it appears better, but soon relapses. There does not appear to be any relationship between visible site activity and the server sluggishness. Whatever is causing this appears to be an internal Plone issue. I say this because while network traffic is minimal, the task manager performance window shows cpu usage continually spiking, often times up to 85-95%, and under processes, directly related to this spiking is the python process (sometimes multiple instances) which is the heavy user/culprit. Quite frankly, I have never seen this kind of continuously high level up and down activity on a web server before.
> 
> Any ideas as to what could be causing this? Crawlers? Spammers? Any thoughts as to what settings we might check, processes to turn off (even temporarily)? Might any one of you be able to take a look at our setup and see what is going on? Recommend someone who could?? We have $$ as needed if we can get some fast help.
> 
> A member of our team not particularly familiar with Plone, but who knows servers well offered this:
> “My first guess is that we are looking at a search engine or internal indexing issue.
> 
> The fact that they have huge slowdowns at some predictable times makes me think "web crawler".  The site is dynamic, so a poorly optimized web crawler could cause major headaches.
> 
> The other possibility is that their internal indexing is problematic.  From the little I've seen of the site configuration previously it looks to me like they try to index just about everything by default with the idea they could turn some off if there are issues.”
> 
> 
> Any suggestions and/or help is greatly appreciated. Thanks.
> 
> 
> Robert Kraus, Web Services
> FPG Child Development Institute
> University of North Carolina at Chapel Hill
> Campus Box 8185
> Chapel Hill, NC 27599-8040
> 
> robert_kraus at unc.edu<mailto:robert_kraus at unc.edu>
> phone: 919.962.7310
> 
> _______________________________________________
> TriZPUG mailing list
> TriZPUG at python.org<mailto:TriZPUG at python.org>
> http://mail.python.org/mailman/listinfo/trizpug
> http://trizpug.org is the Triangle Zope and Python Users Group
> _______________________________________________
> TriZPUG mailing list
> TriZPUG at python.org
> http://mail.python.org/mailman/listinfo/trizpug
> http://trizpug.org is the Triangle Zope and Python Users Group