NumPy frombuffer giving nonsense values when reading C float array on Windows

[eryk sun]

 On Tue, Jul 26, 2016 at 6:31 PM, sth <urschrei at gmail.com> wrote:
>
> The restype is a ctypes Structure instance with a single __fields__ entry (coords), which

Watch the underscores with ctypes attributes. Your code spells it
correctly as "_fields_".

> is a Structure with two fields (len and data) which are the FFI array's length and the void
> pointer to its memory:
> https://github.com/urschrei/pypolyline/blob/master/pypolyline/util.py#L109-L117

_FFIArray.__init__ isn't properly keeping a reference to the wrapped
numpy array:

    def __init__(self, seq, data_type = c_double):
        ptr = POINTER(data_type)
        nparr = np.array(seq, dtype=np.float64)
        arr = nparr.ctypes.data_as(ptr)
        self.data = cast(arr, c_void_p)
        self.len = len(seq)

arr doesn't have a reference to nparr, so self.data doesn't have a
reference to the numpy array when it goes out of scope. For example,
we can trigger a segfault here:

    >>> nparr = np.array(range(2**24), dtype=np.float64)
    >>> ptr = ctypes.POINTER(ctypes.c_double)
    >>> arr = nparr.ctypes.data_as(ptr)
    >>> arr[0], arr[2**24-1]
    (0.0, 16777215.0)

    >>> del nparr
    >>> arr[0], arr[2**24-1]
    Segmentation fault (core dumped)

> I'm only half-following your explanation of how ctypeslib works, but it seems clear that I'm doing
> something wrong.

The library requires the data pointers wrapped in a struct with the
length, so I think what you're doing to wrap and unwrap arbitrary
sequences is generally fine. But you don't need the bytes copy from
string_at. You can cast to a double pointer and create a numpy array
from that, all without copying any data. The only copy made is for
tolist(). For example:

    def _void_array_to_nested_list(res, func, args):
        """ Dereference the FFI result to a list of coordinates """
        try:
            shape = res.coords.len, 2
            ptr = cast(res.coords.data, POINTER(c_double))
            array = np.ctypeslib.as_array(ptr, shape)
            return array.tolist()
        finally:
            drop_array(res.coords)

If you're not hard coding the double data type, consider adding a
simple C type string to the _FFIArray struct. For example:

    >>> ctypes.c_double._type_
    'd'

You just need a dict to map type strings to simple C types.