[Numpy-discussion] Warnings in numpy.ma.test()

[Darren Dale]

On Wed, Mar 17, 2010 at 10:45 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Wed, Mar 17, 2010 at 6:19 AM, Darren Dale <dsdale24 at gmail.com> wrote:
>>
>> On Wed, Mar 17, 2010 at 2:07 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>> > All,
>> > As you're probably aware, the current test suite for numpy.ma raises
>> > some nagging warnings such as "invalid value in ...". These warnings are
>> > only issued when a standard numpy ufunc (eg., np.sqrt) is called on a
>> > MaskedArray, instead of its numpy.ma (eg., np.ma.sqrt) equivalent. The
>> > reason is that the masked versions of the ufuncs temporarily set the numpy
>> > error status to 'ignore' before the operation takes place, and reset the
>> > status to its original value.
>>
>> > I thought I could use the new __array_prepare__ method to intercept the
>> > call of a standard ufunc. After actual testing, that can't work.
>> > __array_prepare only help to prepare the *output* of the operation, not to
>> > change the input on the fly, just for this operation. Actually, you can
>> > modify the input in place, but it's usually not what you want.
>>
>> That is correct, __array_prepare__ is called just after the output
>> array is created, but before the ufunc actually gets down to business.
>> I have the same limitation in quantities you are now seeing with
>> masked array, in my case I want the opportunity to rescale different
>> but compatible quantities for the operation (without changing the
>> original arrays in place, of course).
>>
>> > Then, I tried to use  __array_prepare__ to store the current error
>> > status in the input, force it to ignore divide/invalid errors and send the
>> > input to the ufunc. Doesn't work either: np.seterr in __array_prepare__ does
>> > change the error status, but as far as I understand, the ufunc is called is
>> > still called with the original error status. That means that if something
>> > goes wrong, your error status can stay stuck. Not a good idea either.
>> > I'm running out of ideas at this point. For the test suite, I'd suggest
>> > to disable the warnings in test_fix_invalid and test_basic_arithmetic.
>> > An additional issue is that if one of the error status is set to
>> > 'raise', the numpy ufunc will raise the exception (as expected), while its
>> > numpy.ma version will not. I'll put also a warning in the docs to that
>> > effect.
>> > Please send me your comments before I commit any changes.
>>
>> I started thinking about a third method called __input_prepare__ that
>> would be called on the way into the ufunc, which would allow you to
>> intercept the input and pass a somehow modified copy back to the
>> ufunc. The total flow would be:
>>
>> 1) Call myufunc(x, y[, z])
>> 2) myufunc calls ?.__input_prepare__(myufunc, x, y), which returns x',
>> y' (or simply passes through x,y by default)
>> 3) myufunc creates the output array z (if not specified) and calls
>> ?.__array_prepare__(z, (myufunc, x, y, ...))
>> 4) myufunc finally gets around to performing the calculation
>> 5) myufunc calls ?.__array_wrap__(z, (myufunc, x, y, ...)) and returns
>> the result to the caller
>>
>> Is this general enough for your use case? I haven't tried to think
>> about how to change some global state at one point and change it back
>> at another, that seems like a bad idea and difficult to support.
>>
>
> I'm not a masked array user and not familiar with the specific problems
> here, but as an outsider it's beginning to look like one little fix after
> another.

Yeah, I was concerned that criticism would come up.

> Is there some larger framework that would help here?

I think there is: http://www.python.org/dev/peps/pep-3124/

> Changes to the ufuncs themselves?

Perhaps, if ufuncs were instances of a class that implemented
__call__, it would be easier to include context management. Maybe this
approach could be coupled with input_prepare, array_prepare and
array_wrap to provide everything we need.

> There was some code for masked ufuncs on the c level
> posted a while back that I thought was interesting, would it help to have
> masked masked versions of the ufuncs?

I think we need a solution that avoids implementing an entirely new
set of ufuncs for specific subclasses.

> So on and so forth. It just looks like a larger design issue needs to be addressed here.

I'm interested to hear other people's perspectives or suggestions.

Darren