[Numpy-discussion] Memory layout of record arrays

[Nicolas Rougier]

I've cooked a very rudimentary implementation of what I would like to
have, however I've got a small problem concerning array shape. The idea
is to have a parent array (a group) that can be instantiated like a
regular array and that build internally all the relevant contiguous
child arrays. I would like those child arrays to not be reshapable (at
least not without telling their parent first) and I naively overloaded
the shape property but got a lot of problems with infinite recursion. Is
there a way to do that ?

Nicolas



I put the code below (around 100 lines):

class array(np.ndarray):

    def __new__(subtype, shape=(1,1), dtype=np.float32, order=None,
group=None):
        obj = np.ndarray.__new__(subtype, shape=shape, dtype=dtype,
order=order)
        obj._group = group
        return obj

    def _get_group(self):
        return self._group or self
    def _set_group(self, group):
        if self.size == group.size:
            self.shape = group.shape
            self._group = group
        else:
            raise ValueError, \
                'shape mismatch: objects cannot be broadcast to a single
shape'
    group = property(_get_group, _set_group,
                      doc = '''Group to which this array belongs to''')

    def _get_shape(self):
        return self.ctypes.shape
    def _set_shape(self, shape):
        if self.group == None:
            self.ctypes.shape = shape
        else:
            raise AttributeError, \
                '''Cannot reshape a child array (''group'' is not
None)'''
#    shape = property(_get_shape, _set_shape, doc='''c-types shape''')
    

class group(object):
    def __init__(self, shape, dtype=None, order=None):
        object.__setattr__(self,'_arrays', {})
        self._shape = shape
        self._dtype = dtype        
        if len(dtype) == 0:
            self._dtype = np.dtype('f0',dtype)
        for i in range(len(self._dtype)):
            name, dtype = self._dtype[i]
            self._arrays[name] =
array(shape=shape,dtype=dtype,order=order)


    def __getattr__(self, key):
        if key in self._arrays.keys():
            return self._arrays[key]
        else:
            return object.__getattribute__(self, key)
    def __setattr__(self, key, value):
        if key in self._arrays.keys():
            self._arrays[key][...] = value
        else:
            object.__setattr__(self, key, value)

    def __getitem__(self, key):
        return self._arrays[key]
    def __setitem__(self, key, value):
        self._arrays[key][...] = value
    def __len__(self):
        return len(self._arrays[self._arrays.keys()[0]])


    def _get_shape(self):
        return self._shape
    def _set_shape(self, shape):
        for key in self._arrays.keys():
            self._arrays[key].shape = shape
        self._shape = shape
    shape = property(_get_shape, _set_shape)


    def _get_dtype(self):
        return self._dtype
    def _set_dtype(self):
        raise AttributeError, \
            '''attribute 'dtype' of 'group' objects is not writable'''
    dtype = property(_get_dtype, _set_dtype)


    def _get_size(self):
        return self._arrays[self._arrays.keys()[0]].size
    def _set_size(self):
        raise AttributeError, \
            '''attribute 'size' of 'group' objects is not writable'''
    size = property(_get_size, _set_size)


    def __repr__(self):
        s = 'group(\n'
        for i in range(len(self._dtype)):
            name,dtype = self._dtype[i]            
            t = "'%s': " % name
            a = repr(self._arrays[name]).replace('\n', '\n' +' '*len(t))
            s += t+a+'\n'
        s += ')'
        return s

if __name__ == '__main__':

    G = group((3,3), dtype =
[('r',np.float32),('g',np.int32),('b',np.bool)])
    G['r'] = G.g = G.b = 0
    print G
    print
    G.r.shape = (9,1)
    print G.r.shape
    print G.shape







On Thu, 2009-07-30 at 20:01 +0200, Nicolas Rougier wrote:
> 
> Thanks for the quick answer. It makes sense.
> I will have to find some other way to do it then.
> 
> Nicolas
> 
> 
> On 30 Jul, 2009, at 18:52 , David Cournapeau wrote:
> 
> > On Fri, Jul 31, 2009 at 12:53 AM, Nicolas
> > Rougier<Nicolas.Rougier at loria.fr> wrote:
> >>
> >>
> >> Hello,
> >>
> >> I've been using record arrays to create arrays with different types
> >> and since I'm doing a lot of computation on each of the different
> >> fields, the default memory layout does not serve my computations.
> >> Ideally, I would like to have record arrays where each field is a
> >> contiguous block of memory.
> >
> > I don't think you can do it with record arrays: one of the fundamental
> > design choice of numpy array layout is that the data pointer points to
> > one block of N items, where each item is described by the dtype. To
> > have contiguous layout for each member of the structured dtype, you
> > need two arrays with the corresponding dtype.
> >
> > cheers,
> >
> > David
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion