NumPy frombuffer giving nonsense values when reading C float array on Windows

[sth]

On Tuesday, 26 July 2016 19:10:46 UTC+1, eryk sun  wrote:
> On Tue, Jul 26, 2016 at 12:06 PM,  sth wrote:
> > I'm using ctypes to interface with a binary which returns a void pointer (ctypes c_void_p) to a nested 64-bit float array:
> 
> If this comes from a function result, are you certain that its restype
> is ctypes.c_void_p? I commonly see typos here such as setting
> "restypes" instead of "restype".
> 
> > [[1.0, 2.0], [3.0, 4.0], … ]
> > then return the pointer so it can be freed
> >
> > I'm using the following code to de-reference it:
> >
> > # a 10-element array
> > shape = (10, 2)
> > array_size = np.prod(shape)
> > mem_size = 8 * array_size
> > array_str = ctypes.string_at(ptr, mem_size)
> > # convert to NumPy array,and copy to a list
> > ls = np.frombuffer(array_str, dtype="float64", count=array_size).reshape(shape).tolist()
> > # return pointer so it can be freed
> > drop_array(ptr)
> > return ls
> >
> > This works correctly and consistently on Linux and OSX using NumPy 1.11.0, but fails on
> > Windows 32 bit and 64-bit about 50% of the time, returning nonsense values. Am I doing
> > something wrong? Is there a better way to do this?
> 
> numpy.ctypeslib facilitates working with ctypes functions, pointers
> and arrays via the factory functions as_array, as_ctypes, and
> ndpointer.
> 
> ndpointer creates a c_void_p subclass that overrides the default
> from_param method to allow passing arrays as arguments to ctypes
> functions and also implements the _check_retval_ hook to automatically
> convert a pointer result to a numpy array.
> 
> The from_param method validates an array argument to ensure it has the
> proper data type, shape, and memory layout. For example:
> 
>     g = ctypes.CDLL(None) # Unix only
>     Base = np.ctypeslib.ndpointer(dtype='B', shape=(4,))
> 
>     # strchr example
>     g.strchr.argtypes = (Base, ctypes.c_char)
>     g.strchr.restype = ctypes.c_char_p
> 
>     d = np.array(list(b'012\0'), dtype='B')
>     e = np.array(list(b'0123\0'), dtype='B') # wrong shape
> 
>     >>> g.strchr(d, b'0'[0])
>     b'012'
>     >>> g.strchr(e, b'0'[0])
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     ctypes.ArgumentError: argument 1: <class 'TypeError'>:
>     array must have shape (4,)
> 
> The _check_retval_ hook of an ndpointer calls numpy.array on the
> result of a function. Its __array_interface__ property is used to
> create a copy with the defined data type and shape. For example:
> 
>     g.strchr.restype = Base
> 
>     >>> d.ctypes._as_parameter_ # source address
>     c_void_p(24657952)
>     >>> a = g.strchr(d, b'0'[0])
>     >>> a
>     array([48, 49, 50,  0], dtype=uint8)
>     >>> a.ctypes._as_parameter_ # it's a copy
>     c_void_p(19303504)
> 
> As a copy, the array owns its data:
> 
>     >>> a.flags
>       C_CONTIGUOUS : True
>       F_CONTIGUOUS : True
>       OWNDATA : True
>       WRITEABLE : True
>       ALIGNED : True
>       UPDATEIFCOPY : False
> 
> You can subclass the ndpointer type to have _check_retval_ instead
> return a view of the result (i.e. copy=False), which may be desirable
> for a large result array but probably isn't worth it for small arrays.
> For example:
> 
>     class Result(Base):
>         @classmethod
>         def _check_retval_(cls, result):
>             return np.array(result, copy=False)
> 
>     g.strchr.restype = Result
> 
>     >>> a = g.strchr(d, b'0'[0])
>     >>> a.ctypes._as_parameter_ # it's NOT a copy
>     c_void_p(24657952)
> 
> Because it's not a copy, the array view doesn't own the data, but note
> that it's not a read-only view:
> 
>     >>> a.flags
>       C_CONTIGUOUS : True
>       F_CONTIGUOUS : True
>       OWNDATA : False
>       WRITEABLE : True
>       ALIGNED : True
>       UPDATEIFCOPY : False

The restype is a ctypes Structure instance with a single __fields__ entry (coords), which is a Structure with two fields (len and data) which are the FFI array's length and the void pointer to its memory:
https://github.com/urschrei/pypolyline/blob/master/pypolyline/util.py#L109-L117

I'm only half-following your explanation of how ctypeslib works, but it seems clear that I'm doing something wrong.