[SciPy-User] Large Memory usage while doing median filter

[Warren Weckesser]

On Mon, May 11, 2015 at 10:34 AM, Moore, Eric (NIH/NIDDK) [F] <
eric.moore2 at nih.gov> wrote:

> From: Warren Weckesser [mailto:warren.weckesser at gmail.com]
> Sent: Monday, May 11, 2015 9:54 AM
> To: SciPy Users List
> Subject: Re: [SciPy-User] Large Memory usage while doing median filter
>
>
>
> On Mon, May 11, 2015 at 1:25 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr>
> wrote:
> On Mon, 11 May 2015 00:56:29 +0530
> Joe P Ninan <indiajoe at gmail.com> wrote:
>
> > Hi,
> > I was trying median_filter in scipy.ndimage.filters
> > on a 1024x1024 array.
> >
> > What I noticed is that the memory requirement grows really fast when we
> > increase the size of the median filter.
> > On a machine with 6gb RAM I could do only (150,150) size filter.
> > Anything above gives Memory Error.
> >
> > On a bigger server I could see it takes about 16gb RAM while using a
> filter
> > size (200, 200)
> >
> > I can understand, computation time increasing with size of filter, but
> why
> > is the memory size exploding with respect to size of the median filter?
> > Is this expected behaviour?
>
> I guess this is because scipy creates a 1024x1024x(40000) array to do the
> sort along the last axis.
> maybe no the best from the memorry point of view.
>
> Maybe I didn't search hard enough, but I don't see where such an array is
> allocated.  There are several layers of calls, from python in
> ndimage/filters.py down to C in ndimage/src/ni_filters.c, so maybe I missed
> it.  Can you point to where such an array is created, or was that really a
> guess?
> Warren
>
> The really large array is allocated in NI_InitFilterOffsets, on line 518,
> in ni_support.c which is called from line 726 of ni_filter.c, in
> Ni_RankFilter.
>


Thanks Eric.

Warren



>
> For me, calling ndimage.median_filter(arr, 150), with arr a (1024, 1024)
> array of doubles or floats results in an allocation of 4050000000 bytes (
> 3.77 GB).  Which seems a little bit bigger than we would like here.
>
> -Eric
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20150511/35aa081e/attachment.html>