script uses up all memory

[Larry Martell]

On Thu, Mar 6, 2014 at 5:11 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Fri, Mar 7, 2014 at 8:56 AM, Larry Martell <larry.martell at gmail.com> wrote:
>> I figured out what is causing this. Each pass through the loop it does:
>>
>> self.tools = Tool.objects.filter(ip__isnull=False)
>>
>> And that is what is causing the memory consumption. If I move that
>> outside the loop and just do that once the memory issue goes away. Now
>> I need to figure out why this is happening and how to prevent it as
>> they do want to query the db each pass through the loop in case it has
>> been updated.
>
> Interesting. So the next thing to do is to look into the
> implementation of that. Does it allocate database resources and not
> free them? Does it have internal reference loops?
>
> Something to try: Put an explicit gc.collect() call into the loop. If
> that solves your problem, you have a refloop somewhere (and you can
> properly fix it by explicitly breaking the loop). If that keeps
> returning large numbers, and especially if it populates gc.garbage
> with a whole lot of stuff, then you definitely have refloops.
>
> http://docs.python.org/2/library/gc.html


First I added del(self.tools) before the Django call. That did not
stop the memory consumption. Then I added a call to gc.collect() after
the del and that did solve it. gc.collect() returns 0 each time, so
I'm going to declare victory and move on. No time to dig into the
Django code. Thanks.