[SciPy-User] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence"

[Nathaniel Smith]

[CC'ing scipy-dev because see below]

On Tue, Sep 24, 2013 at 6:45 PM, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> I'm getting a very strange error, and unfortunately I can't seem to
> reproduce it *except* when running on Travis-CI, so my debugging tools
> are really limited. I'm wondering if anyone else has seen anything
> like this?
>
> The offending line of code is:
>
>   File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pyrerp/incremental_ls.py",
> line 323, in append_bottom_half
>     self.xtx += xtx
> ValueError: setting an array element with a sequence.
>
> And debug prints reveal that in the case that causes the error,
> 'self.xtx' is an ndarray that prints as:
>
> [[ 0.  0.]
>  [ 0.  0.]]
>
> while 'xtx' is a scipy.sparse.csc.csc_matrix that prints as:
>
>   (1, 0) 45.0
>   (0, 0) 10.0
>   (1, 1) 285.0
>   (0, 1) 45.0

In accordance with the cosmic law governing such things, this turns
out to be a bug in numpy that I introduced in 2397c9d4, specifically
this line, which introduces an unchecked DEPRECATE:
  https://github.com/numpy/numpy/commit/2397c9d4#L5R528
(Plus some complicated nonsense involving virtualenvs that sometimes
fall back on system libraries even though the virtualenv contains a
perfectly good version of the same library etc. to make reproduction
more confusing.)

My test code was setting warnings to raise errors by default, and
apparently ndarray += csc_matrix goes through some circuitous path
that (AFAICT) involves creating an object ndarray containing the
csc_matrix and calling the add ufunc, which somehow trips on the cast
warning above.

Then later on, at line 159 of arraytypes.c.src, the the @TYPE at _setitem
function does:

    if (PyErr_Occurred()) {
        if (PySequence_Check(op) && !PyString_Check(op) &&
                                            !PyUnicode_Check(op)) {
            PyErr_Clear();
            PyErr_SetString(PyExc_ValueError,
                    "setting an array element with a sequence.");
        }
        return -1;
    }

and the PyErr_Occurred here catches the real error and replaces it
with some nonsense.

So three points:
1) Maybe this will be useful to someone googling later.
2) I guess I'll file some horrible patch against numpy that just
throws away exceptions caused by that deprecation, because there is no
way for PyArray_CanCastTypeTo to raise an error :-(.
3) This script raises a DeprecationWarning with numpy 1.7.1 and scipy 0.12.0:

import warnings, numpy, scipy.sparse
warnings.filterwarnings("always")
a = numpy.zeros((2, 2))
b = scipy.sparse.csc_matrix([[0.0, 0.0], [0.0, 0.0]])
a += b

test-script3.py:5: DeprecationWarning: Implicitly casting between
incompatible kinds. In a future numpy release, this will raise an
error. Use casting="unsafe" if this is intentional.

I really don't understand what arcane magic is used to make ndarray +=
csc_matrix work at all, but my question is, is it going to break when
we complete the casting transition described above? It was just
supposed to catch things like int += float.

-n