[Numpy-discussion] [ANN] Nanny, faster NaN functions

[Keith Goodman]

On Fri, Nov 19, 2010 at 12:29 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen <pav at iki.fi> wrote:
>> Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote:
>> [clip]
>>> My guess is that having separate underlying functions for each dtype,
>>> ndim, and axis would be a nightmare for a large project like Numpy. But
>>> manageable for a focused project like nanny.
>>
>> Might be easier to migrate the nan* functions to using Ufuncs.
>>
>> Unless I'm missing something,
>>
>>        np.nanmax -> np.fmax.reduce
>>        np.nanmin -> np.fmin.reduce
>>
>> For `nansum`, we'd need to add an ufunc `nanadd`, and for
>> `nanargmax/min`, we'd need `argfmin/fmax'.
>
> How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, please.
>
>>> arr = np.random.rand(1000, 1000)
>>> arr[arr > 0.5] = np.nan
>>> np.nanmax(arr)
>   0.49999625409581072
>>> np.fmax.reduce(arr, axis=None)
> <snip>
> TypeError: an integer is required
>>> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0)
>   0.49999625409581072
>
>>> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0)
> 100 loops, best of 3: 12.7 ms per loop
>>> timeit np.nanmax(arr)
> 10 loops, best of 3: 39.6 ms per loop
>
>>> timeit np.nanmax(arr, axis=0)
> 10 loops, best of 3: 46.5 ms per loop
>>> timeit np.fmax.reduce(arr, axis=0)
> 100 loops, best of 3: 12.7 ms per loop

Cython is faster than np.fmax.reduce.

I wrote a cython version of np.nanmax, called nanmax below. (It only
handles the 2d, float64, axis=None case, but since the array is large
I don't think that explains the time difference).

Note that fmax.reduce is slower than np.nanmax when there are no NaNs:

>> arr = np.random.rand(1000, 1000)
>> timeit np.nanmax(arr)
100 loops, best of 3: 5.82 ms per loop
>> timeit np.fmax.reduce(np.fmax.reduce(arr))
100 loops, best of 3: 9.14 ms per loop
>> timeit nanmax(arr)
1000 loops, best of 3: 1.17 ms per loop

>> arr[arr > 0.5] = np.nan

>> timeit np.nanmax(arr)
10 loops, best of 3: 45.5 ms per loop
>> timeit np.fmax.reduce(np.fmax.reduce(arr))
100 loops, best of 3: 12.7 ms per loop
>> timeit nanmax(arr)
1000 loops, best of 3: 1.17 ms per loop