python gc performance in large apps

[Bryan Olson]

Robby Dermody wrote:
> [...] However, on a 
> simulated call center environment (constant 50 conversations, switching 
> every 300 seconds) the director component still consumes an average of 
> 1MB more per hour and the harvester is taking an average of 4MB more per 
> hour. With the director, 2/3 of this is resident (1/3 in swap). With the 
> harvester, about 88% of this is resident (~ 12% in swap).

For a long-uptime server, that's obviously not going to work.
Memory leaks are of course, bugs, and bugs call for debugging,
but given the long history of such bugs in Python and similar
projects, there's no real chance of eliminating them.

You might look at how the Apache HTTP server deals with this.
Apache can stay up arbitrarily long, even when it links in
modules with slow memory leaks. A parent process forks off
child processes as needed, and the children handle all the
client services. Each child is, by default, limited to
serving 10,000 connections, a number adjustable via the
MaxRequestsPerChild directive. When a child reaches its
connection limit, it stops listening for new connection
requests; when the last of the connection closes, the
child gracefully dies.


 > [...] I doubt we can remove all the memory
 > problems, but if we can get this down by another 150 - 200%, that will
 > be good enough for me.

Down by more than 100% ?  Wouldn't that require turning
memory leaks into memory wells?


-- 
--Bryan