[Chicago] Out of Memory: Killed Process: on CentOS

[Garrett Smith]

----- "Brian Ray" <bray at sent.com> wrote:
> Hello:
> 
> I am trying to tackle an issue I am having with httpd running an app 
> under mod_python not being able to kill enough processes (to free  
> memory) quickly enough to remain stable.  The server ends up in a  
> state where it stops responding.

>From the configuration you're describing, this could be a bunch of
things.

> The server has several Gig's of memory, but the requests it tries to 
> process are really huge.  It run's fine if the requests are small.

What do you mean by big/small? Size of HTTP request bodies?

> What is the best ways to manage this scenario?  Specifically, what  
> should I try?

I suggest first confirming that it's memory. Get the server into the
non responsive state and use 'top' to survey the problematic processes.
By default, top will list the top-CPU consumers first. Use SHIFT + >
(greater than sign) to change the sort order to list the top memory
consumers.

top also shows uptime stats on the top line. Big numbers probably mean
lots of processes running at full tilt. This would cause server slow
down.

For memory, you're interested in RSS (resident) - that's the memory the
process is actually using (as opposed to what's reserved for it).

You can install sysstat and run pidstat to track a particular process,
but I suspect top is all you'll need.

> Should I be looking at my apache configuration?  I tried setting  
> StartServers, MinSpareServers, MaxSpareServers, and MaxClients to low 
> numbers, like 3.  Overall it seems to help but, I am not sure it will 
> if I get a really huge request.

I'd recommend running your Python app as a standalone, multithreaded
server (e.g. use CherryPy). This is going to eliminate a huge number
of variables and help you isolate problems more quickly.

Take some time and get into the details of apache + mod_python before
running apps in production. I'd offer some specific suggestions, but
it's been a while since I've used that mod -- and it's got a lot of
configuration options.

You might also look into modwsgi -- this has really matured over the
last couple years. If I use apache, I generally use that mod for Python
apps.

> Is there some other way to clear memory when running large requests?

Verify that you understand what's causing your server to become
unresponsive before getting into application level tweaks. If it is
indeed memory, first check for leaks (in Python, best case is global
state including static/class variables, caches that aren't being
attended to, etc.) You'll see this right away if you run as a standalone
app -- you're process will keep gobbling memory (use top to watch) and
the server will eventually start swapping.

If it's simply that cElementTree is keeping huge structures in memory
and you're getting too many concurrent requests for your RAM, move to
a SAX style API (state machine).

Garrett