why does memory consumption keep growing?

[Steve D'Aprano]

On Fri, 6 Oct 2017 08:06 am, Fetchinson . wrote:

> Hi folks,
> 
> I have a rather simple program which cycles through a bunch of files,
> does some operation on them, and then quits. There are 500 files
> involved and each operation takes about 5-10 MB of memory. As you'll
> see I tried to make every attempt at removing everything at the end of
> each cycle so that memory consumption doesn't grow as the for loop
> progresses, but it still does.

How do you know memory consumption is still growing?

I'm not saying it isn't, but knowing what the symptoms are and how you
determined the memory leak may be important in solving it.


> import os
> for f in os.listdir( '.' ):
>     x = [ ]
>     for ( i, line ) in enumerate( open( f ) ):
>         import mystuff
>         x.append( mystuff.expensive_stuff( line ) )
>         del mystuff

As Chris has already said, deleting and re-importing mystuff is a waste of
time.


>     import mystuff
>     mystuff.some_more_expensive_stuff( x )
>     del mystuff
>     del x
> 
> 
> What can be the reason? I understand that mystuff might be leaky,

You know it is, you have good reason to think it is, or you're just guessing?


> but 
> if I delete it, doesn't that mean that whatever memory was allocated
> is freed? Similary x is deleted so that can't possibly make the memory
> consumption go up.

You are not deleting the list. You are deleting a single reference to the
list, which may or may not allow the list to be garbage collected.

I hesitate to suggest this, because I don't want to encourage superstitious
cargo-cult programming, but:

(1) assuming you have good evidence that memory consumption is increasing;

(2) rather than `del x`, try `x[:] = []`;

(3) then if memory consumption stops increasing

that may be evidence of a leak, and provide a reasonable work around until the
leak can be fixed.

But in a well-behaved program, you shouldn't need to manually delete x at all.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.