[Numpy-discussion] PyArray_SimpleNewFromData segfaults

[Ondrej Certik]

On Mon, Oct 5, 2009 at 9:25 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Mon, Oct 5, 2009 at 8:38 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>> On Mon, Oct 5, 2009 at 7:34 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>>
>>>
>>> On Mon, Oct 5, 2009 at 7:40 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>> [...]
>>>> still alive
>>>> Segmentation fault
>>>>
>>>>
>>>>
>>>> What puzzles me is that there is no debugging print statement just
>>>> before the segfault.
>>>
>>> Maybe you need to flush the buffer. That is a good thing to do when
>>> segfaults are about.
>>
>> I tried to put "fflush(NULL);" after it, but it didn't help. I have
>> created a super simple demo for anyone to play:
>>
>>
>> $ git clone git://github.com/certik/segfault.git
>> $ cd segfault/
>> $ vim Makefile     # <-- edit the python and numpy include paths
>> $ make
>> $ python test.py
>> I am still alive
>> Segmentation fault
>>
>> where test.py is:
>>
>> $ cat test.py
>> import _hermes1d
>> v = _hermes1d.test()
>> print v
>>
>>
>> and _hermes1d.pyx is:
>>
>> $ cat _hermes1d.pyx
>> def test():
>>    cdef npy_intp size
>>    cdef ndarray newarr
>>    cdef double *arrsource
>>
>>    size = 10
>>    arrsource = <double *>malloc(sizeof(double) * size)
>>    print "I am still alive"
>>    newarr = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, <void *>arrsource)
>>    print "I am dead."
>>
>>    return newarr
>>
>>
>> So I bet there is something very stupid that I am missing. Still
>> investigating...
>
> I didn't call _import_array()  !
>
> This patch fixes it:
>
>
> diff --git a/_hermes1d.pxd b/_hermes1d.pxd
> index 9994c28..f5e8868 100644
> --- a/_hermes1d.pxd
> +++ b/_hermes1d.pxd
> @@ -54,6 +54,8 @@ cdef extern from "arrayobject.h":
>     object PyArray_SimpleNewFromData(int nd, npy_intp* dims, int typenum,
>             void* data)
>
> +    void _import_array()
> +
>  cdef extern from "Python.h":
>     ctypedef void PyObject
>     void Py_INCREF(PyObject *x)
> diff --git a/_hermes1d.pyx b/_hermes1d.pyx
> index e542ddc..7a4beec 100644
> --- a/_hermes1d.pyx
> +++ b/_hermes1d.pyx
> @@ -2,6 +2,7 @@ def test():
>     cdef npy_intp size
>     cdef ndarray newarr
>     cdef double *arrsource
> +    _import_array()
>
>     size = 10
>     arrsource = <double *>malloc(sizeof(double) * size)
>
>
>
>
> I think I learned something today the hard way.


The only mention of the _import_array() in the documentation that I
found is here:

http://docs.scipy.org/doc/numpy/reference/c-api.array.html#NO_IMPORT_ARRAY


but I don't understand what it means ---- do I have to just call
_import_array() and then I can use numpy CAPI, or do I also have to
define those PY_ARRAY_UNIQUE_SYMBOL etc?


Btw, to explain my original post for future readers --- the real
problem was that PyArray_Type was NULL and thus &PyArray_Type
segfaulted. That happened in the definition:

#define PyArray_SimpleNewFromData(nd, dims, typenum, data)                    \
       PyArray_New(&PyArray_Type, nd, dims, typenum, NULL,                   \
                   data, 0, NPY_CARRAY, NULL)

so it is *extremely* confusing, since PyArray_SimpleNewFromData() was
being called from my code, but PyArray_New never started, it
segfaulted in between.

I think now it is clear what is going on. Only I don't understand the
intention, but I can now get my job done.

Ondrej