[SciPy-User] Large Memory usage while doing median filter

[Moore, Eric (NIH/NIDDK) [F]]

From: Warren Weckesser [mailto:warren.weckesser at gmail.com] 
Sent: Monday, May 11, 2015 9:54 AM
To: SciPy Users List
Subject: Re: [SciPy-User] Large Memory usage while doing median filter



On Mon, May 11, 2015 at 1:25 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr> wrote:
On Mon, 11 May 2015 00:56:29 +0530
Joe P Ninan <indiajoe at gmail.com> wrote:

> Hi,
> I was trying median_filter in scipy.ndimage.filters
> on a 1024x1024 array.
>
> What I noticed is that the memory requirement grows really fast when we
> increase the size of the median filter.
> On a machine with 6gb RAM I could do only (150,150) size filter.
> Anything above gives Memory Error.
>
> On a bigger server I could see it takes about 16gb RAM while using a filter
> size (200, 200)
>
> I can understand, computation time increasing with size of filter, but why
> is the memory size exploding with respect to size of the median filter?
> Is this expected behaviour?

I guess this is because scipy creates a 1024x1024x(40000) array to do the sort along the last axis.
maybe no the best from the memorry point of view.

Maybe I didn't search hard enough, but I don't see where such an array is allocated.  There are several layers of calls, from python in ndimage/filters.py down to C in ndimage/src/ni_filters.c, so maybe I missed it.  Can you point to where such an array is created, or was that really a guess?
Warren

The really large array is allocated in NI_InitFilterOffsets, on line 518, in ni_support.c which is called from line 726 of ni_filter.c, in Ni_RankFilter.

For me, calling ndimage.median_filter(arr, 150), with arr a (1024, 1024) array of doubles or floats results in an allocation of 4050000000 bytes ( 3.77 GB).  Which seems a little bit bigger than we would like here.

-Eric