script uses up all memory

[Chris Angelico]

On Thu, Mar 6, 2014 at 11:20 AM, Larry Martell <larry.martell at gmail.com> wrote:
> On Wed, Mar 5, 2014 at 5:39 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell <larry.martell at gmail.com> wrote:
>>> I have a script that forks off other processes and attempts to manage
>>> them. Here is a stripped down version of the script:
>>>
>>>         self.sleepTime = 300
>>
>> That's not a stand-alone script.
>
> No, that is just the part that does the work (inside the 'while
> true'). I'll try and post a standalone script tomorrow.
>
>> What environment is it running in?
>
> CentOS 6,4

That's not the whole environment, though. There's a mention of Django
- does this run inside some framework?

>> Can you reproduce the problem outside of that environment?
>
> I will try that tomorrow.

Running as a stand-alone script, still under CentOS, would be what I
mean by "outside of that environment". I'm talking about making
something that can be saved to my drive and executed, perhaps with a
stubby subprocess script (eg "import time; time.sleep(86400)").

>> Also: Can you simply use multiprocessing rather than going through all
>> the effort of subprocess.Popen?
>
> Perhaps. I didn't write this. A client gave it to me and said 'figure
> out why it uses up all the memory and hangs.' I've messed around with
> for days and cannot see anything that would consume so much memory.

Ah. Yeah, that would be a fun little job to play with. My random
thought: Do the subprocesses produce spammy log output? If so, the
monitor might be collecting it all and holding it in memory in case
you want it (not knowing that you don't). The default should be to
leave them connected to your process's stdio streams, though, so that
shouldn't be the issue.

ChrisA