script uses up all memory

[Larry Martell]

On Wed, Mar 5, 2014 at 5:27 PM, Larry Martell <larry.martell at gmail.com> wrote:
> I have a script that forks off other processes and attempts to manage
> them. Here is a stripped down version of the script:
>
>         self.sleepTime = 300
>         self.procs = {}
>         self.startTimes = {}
>         self.cmd = ['python', '/usr/local/motor/motor/app/some_other_script.py']
>
>         while True:
>             try:
>                 self.tools = Tool.objects.filter(ip__isnull=False)
>             except Exception, e:
>                 print 'error from django call: ' + str(e)
>                 sys.exit(1)
>
>             for tool in self.tools:
>                 name = tool.name
>                 if name in self.procs:
>                     if self.procs[name].poll() is None:
>                         if
> (datetime.datetime.now()-self.startTimes[name]) >
> datetime.timedelta(hours=12):
>                             # it's been running too long - kill it
>                             print 'killing script for ' + name + "
> it's been running too long"
>                             self.procs[name].kill()
>                         else:
>                             continue
>
>                     if self.procs[name].returncode:
>                         print 'scrikpt failed for ' + name + ', error
> = ' + str(self.procs[name].returncode)
>
>                     print 'starting script.py for ' + name + ' at ' +
> str(datetime.datetime.now())
>                     try:
>                         self.procs[name] = subprocess.Popen(self.cmd)
>                         self.startTimes[name] = datetime.datetime.now()
>                     except Exception, e:
>                         print 'error from Popen: ' + str(e)
>                         sys.exit(1)
>                 else:
>                     print 'starting script.py for ' + name + ' at ' +
> str(datetime.datetime.now())
>                     try:
>                         self.procs[name] = subprocess.Popen(self.cmd)
>                         self.startTimes[name] = datetime.datetime.now()
>                     except Exception, e:
>                         print 'error from Popen: ' + str(e)
>                         sys.exit(1)
>
>                 time.sleep(self.sleepTime)
>
>
> The script does what it's intended to do, however after about 2 hours
> it has used up all the memory available and the machine hangs. Can
> anyone see something that I am doing here that would be using memory
> like this? Perhaps some immutable object needs to be repeatedly
> recreated?

I figured out what is causing this. Each pass through the loop it does:

self.tools = Tool.objects.filter(ip__isnull=False)

And that is what is causing the memory consumption. If I move that
outside the loop and just do that once the memory issue goes away. Now
I need to figure out why this is happening and how to prevent it as
they do want to query the db each pass through the loop in case it has
been updated.