[SciPy-User] Large Memory usage while doing median filter

[Warren Weckesser]

On Mon, May 11, 2015 at 1:25 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr>
wrote:

> On Mon, 11 May 2015 00:56:29 +0530
> Joe P Ninan <indiajoe at gmail.com> wrote:
>
> > Hi,
> > I was trying median_filter in scipy.ndimage.filters
> > on a 1024x1024 array.
> >
> > What I noticed is that the memory requirement grows really fast when we
> > increase the size of the median filter.
> > On a machine with 6gb RAM I could do only (150,150) size filter.
> > Anything above gives Memory Error.
> >
> > On a bigger server I could see it takes about 16gb RAM while using a
> filter
> > size (200, 200)
> >
> > I can understand, computation time increasing with size of filter, but
> why
> > is the memory size exploding with respect to size of the median filter?
> > Is this expected behaviour?
>
> I guess this is because scipy creates a 1024x1024x(40000) array to do the
> sort along the last axis.
> maybe no the best from the memorry point of view.
>


Maybe I didn't search hard enough, but I don't see where such an array is
allocated.  There are several layers of calls, from python in
ndimage/filters.py down to C in ndimage/src/ni_filters.c, so maybe I missed
it.  Can you point to where such an array is created, or was that really a
guess?

Warren




Cheers,
>
> --
> Jérôme Kieffer
> Data analysis unit - ESRF
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20150511/97ef500f/attachment.html>