[SciPy-Dev] memory leak in scipy.fftpack.ifft2?

[David Warde-Farley]

On Sat, Mar 24, 2012 at 10:22 PM, Brian Toby <brian.toby at anl.gov> wrote:
> Matt Terry <matt.terry <at> gmail.com> writes:
>>
>> I'm assuming that you are expecting the address of CC to remain
>> constant.  As written, it should not.  fft2 returns a new array, as
>> does ifftshift and ifft2.  You can fill an existing array with the
>> answer by creating ffta and CC outside the loop and then filling them
>> with the CC[:,:] = blah() syntax.  The modified script is attached.
>>
>> WIth these changes, the script still uses a lot of memory (high water
>> mark of 2.4 GB), but the memory usage does not grow without bound.  At
>> least for me, using the same platform (mac, epd 7.2).

> That is a very nice trick to force reuse of memory, but it makes even more clear
> there is a memory leak in scipy.fftpack. I was expecting in my previous code
> that python would garbage collect and delete unreferenced objects, but with your
> change arrays are reused so even that is not needed.

I feel I should point out that they are only "reused" up to a point:
sf.ifft2 is here returning an array *that ifft2 is allocating*. CC[:,
:] = sf.ifft2(CC) will copy the contents of that newly allocated array
into the array currently referenced by CC, but the array allocated by
ifft2 will need to be garbage collected before that memory is freed.
The code can't "know" that it's output array is going to be the LHS of
that expression, because the Python interpreter has no way of doing
that kind of introspection. (The way to do this in your own code if
you don't want memory allocated is to pass in an output array.)

You can force a garbage collection at every iteration by sticking
"gc.collect()" in the loop.

I see that fft2 and ifft2 has an "overwrite_x" parameter, which is
what you actually want, but it is *quite* broken (normally these
things only work with Fortran-contiguous inputs, but this isn't
working at all:

>>> a = numpy.array(numpy.random.randn(2, 2), order='F')
>>> a
array([[ 0.18671055, -1.01763466],
       [-0.40909016, -0.43029087]])
>>> a.flags
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
>>> b = scipy.fftpack.fft2(a, overwrite_x=True)
>>> b is a
False
>>> b.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
>>> a
array([[ 0.18671055, -1.01763466],
       [-0.40909016, -0.43029087]])