[Numpy-discussion] Rolling window (moving average, moving std, and more)

[Keith Goodman]

On Tue, Jan 4, 2011 at 8:14 AM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Tue, Jan 4, 2011 at 8:06 AM, Sebastian Haase <seb.haase at gmail.com> wrote:
>> On Mon, Jan 3, 2011 at 5:32 PM, Erik Rigtorp <erik at rigtorp.com> wrote:
>>> On Mon, Jan 3, 2011 at 11:26, Eric Firing <efiring at hawaii.edu> wrote:
>>>> Instead of calculating statistics independently each time the window is
>>>> advanced one data point, the statistics are updated.  I have not done
>>>> any benchmarking, but I expect this approach to be quick.
>>>
>>> This might accumulate numerical errors. But could be fine for many applications.
>>>
>>>> The code is old; I have not tried to update it to take advantage of
>>>> cython's advances over pyrex.  If I were writing it now, I might not
>>>> bother with the C level at all; it could all be done in cython, probably
>>>> with no speed penalty, and maybe even with reduced overhead.
>>>>
>>>
>>> No doubt this would be faster, I just wanted to offer a general way to
>>> this in NumPy.
>>> _______________________________________________
>>
>> BTW, some of these operations can be done using scipy's ndimage  - right ?
>> Any comments ?  How does the performance compare ?
>> ndimage might have more options regarding edge handling, or ?
>
> Take a look at the moving window function in the development version
> of the la package:
>
> https://github.com/kwgoodman/la/blob/master/la/farray/mov.py
>
> Many of the moving window functions offer three calculation methods:
> filter (ndimage), strides (the strides trick discussed in this
> thread), and loop (a simple python loop).
>
> For example:
>
>>> a = np.random.rand(500,2000)
>>> timeit la.farray.mov_max(a, window=252, axis=-1, method='filter')
> 1 loops, best of 3: 336 ms per loop
>>> timeit la.farray.mov_max(a, window=252, axis=-1, method='strides')
> 1 loops, best of 3: 609 ms per loop
>>> timeit la.farray.mov_max(a, window=252, axis=-1, method='loop')
> 1 loops, best of 3: 638 ms per loop
>
> No one method is best for all situations. That is one of the reasons I
> started the Bottleneck package. I figured Cython could beat them all.

I added four new function to Bottleneck: move_min, move_max,
move_nanmin, move_nanmax. They are much faster than using SciPy's
ndimage.maximum_filter1d or the strides trick:

>> a = np.random.rand(500,2000)
>> timeit la.farray.mov_max(a, window=252, axis=-1, method='filter') # ndimage
1 loops, best of 3: 336 ms per loop
>> timeit bn.move_max(a, window=252, axis=-1) # bottleneck
100 loops, best of 3: 14.1 ms per loop

That looks too good to be true. Are the outputs the same?

>> a1 = la.farray.mov_max(a, window=252, axis=-1, method='filter')
>> a2 = bn.move_max(a, window=252, axis=-1)
>> np.testing.assert_array_almost_equal(a1, a2)
>>

Yes.