[SciPy-User] Large Memory usage while doing median filter

[Moore, Eric (NIH/NIDDK) [F]]

From: Joe P Ninan [mailto:indiajoe at gmail.com] 
Sent: Monday, May 11, 2015 9:14 AM
To: SciPy Users List
Subject: Re: [SciPy-User] Large Memory usage while doing median filter

Hi Juan,
Thank you for the suggestion, but my data is 32 bit float. And since the precision in data is important, I cannot convert them to uint8 data.

As Jerome suggested, it might be due to the extra large array scipy is creating to do faster sorting.
In typical astronomy applications I encounter, our images are bigger than 1kx1k, I wonder whether there exist other tools to do median filtering. 
For a moving window median, since only a few pixels leaves and enter the window, if we take advantage of that, then I would imagine the sort time required to find median in each window position wouldn't be very high.

Does anybody know of any such fast median filter routines in python?
Thanking you,
-cheers
joe


On 11 May 2015 at 11:01, Juan Nunez-Iglesias <jni.soma at gmail.com> wrote:
If you can cast your image as a uint8 image, try the median filter in scikit-image's filters.rank module. It's very fast and has a minimal memory footprint. But it doesn't work on floats or high ints. 

—
Sent from Mailbox 

On Mon, May 11, 2015 at 3:25 PM, Jerome Kieffer <Jerome.Kieffer at esrf.fr> wrote:
On Mon, 11 May 2015 00:56:29 +0530 
Joe P Ninan <indiajoe at gmail.com> wrote: 

> Hi, 
> I was trying median_filter in scipy.ndimage.filters 
> on a 1024x1024 array. 
> 
> What I noticed is that the memory requirement grows really fast when we 
> increase the size of the median filter. 
> On a machine with 6gb RAM I could do only (150,150) size filter. 
> Anything above gives Memory Error. 
> 
> On a bigger server I could see it takes about 16gb RAM while using a filter 
> size (200, 200) 
> 
> I can understand, computation time increasing with size of filter, but why 
> is the memory size exploding with respect to size of the median filter? 
> Is this expected behaviour? 

I guess this is because scipy creates a 1024x1024x(40000) array to do the sort along the last axis. 
maybe no the best from the memorry point of view. 
Cheers, 

-- 
Jérôme Kieffer 
Data analysis unit - ESRF 
_______________________________________________ 
SciPy-User mailing list 
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user 


_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



-- 
/---------------------------------------------------------------
"GNU/Linux: because a PC is a terrible thing to waste" -  GNU Generation

************************************************
Joe Philip Ninan       
Research Scholar       
DAA,  TIFR,                           
Mumbai, India.      
Ph: +917738438212  
------------------------------------------------------------
Website: www.tifr.res.in/~ninan/
My GnuPG Public Key: www.tifr.res.in/~ninan/JPN_public.key


Hi Joe,

Would you report this as an issue on github so that it doesn't get lost?  

A second thought is that a different implementation of a median filter exists in the signal package as medfilt and medfilt2d.  I haven't ever used any of these functions, but it might be worth a shot to try them.  

-Eric