[SciPy-User] Large Memory usage while doing median filter

[Joe P Ninan]

Hi Juan,
Thank you for the suggestion, but my data is 32 bit float. And since the
precision in data is important, I cannot convert them to uint8 data.

As Jerome suggested, it might be due to the extra large array scipy is
creating to do faster sorting.
In typical astronomy applications I encounter, our images are bigger than
1kx1k, I wonder whether there exist other tools to do median filtering.
For a moving window median, since only a few pixels leaves and enter the
window, if we take advantage of that, then I would imagine the sort time
required to find median in each window position wouldn't be very high.

Does anybody know of any such fast median filter routines in python?
Thanking you,
-cheers
joe


On 11 May 2015 at 11:01, Juan Nunez-Iglesias <jni.soma at gmail.com> wrote:

> If you can cast your image as a uint8 image, try the median filter in
> scikit-image's filters.rank module. It's very fast and has a minimal memory
> footprint. But it doesn't work on floats or high ints.
>
> —
> Sent from Mailbox <https://www.dropbox.com/mailbox>
>
>
> On Mon, May 11, 2015 at 3:25 PM, Jerome Kieffer <Jerome.Kieffer at esrf.fr>
> wrote:
>
>> On Mon, 11 May 2015 00:56:29 +0530
>> Joe P Ninan <indiajoe at gmail.com> wrote:
>>
>> > Hi,
>> > I was trying median_filter in scipy.ndimage.filters
>> > on a 1024x1024 array.
>> >
>> > What I noticed is that the memory requirement grows really fast when we
>> > increase the size of the median filter.
>> > On a machine with 6gb RAM I could do only (150,150) size filter.
>> > Anything above gives Memory Error.
>> >
>> > On a bigger server I could see it takes about 16gb RAM while using a
>> filter
>> > size (200, 200)
>> >
>> > I can understand, computation time increasing with size of filter, but
>> why
>> > is the memory size exploding with respect to size of the median filter?
>> > Is this expected behaviour?
>>
>> I guess this is because scipy creates a 1024x1024x(40000) array to do the
>> sort along the last axis.
>> maybe no the best from the memorry point of view.
>> Cheers,
>>
>> --
>> Jérôme Kieffer
>> Data analysis unit - ESRF
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
/---------------------------------------------------------------
"GNU/Linux: because a PC is a terrible thing to waste" -  GNU Generation

************************************************
Joe Philip Ninan
Research Scholar
DAA,  TIFR,
Mumbai, India.
Ph: +917738438212
------------------------------------------------------------
Website: www.tifr.res.in/~ninan/ <http://www.tifr.res.in/%7Eninan/>
My GnuPG Public Key: www.tifr.res.in/~ninan/JPN_public.key
<http://www.tifr.res.in/%7Eninan/JPN_public.key>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20150511/24354936/attachment.html>