[Numpy-discussion] resizeable arrays using shared memory?

[Elliot Hallmark]

That makes sense.  I could either send a signal to the child process
letting it know to re-instantiate the numpy array using the same (but now
resized) buffer, or I could have it check to see if the buffer has been
resized when it might need it and re-instantiate then.  That's actually not
too bad.  It would be nice if the array could be resized, but it's probably
unstable to do so and there isn't much demand for it.

Thanks,
  Elliot

On Sat, Feb 6, 2016 at 8:01 PM, Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> On Sa, 2016-02-06 at 16:56 -0600, Elliot Hallmark wrote:
> > Hi all,
> >
> > I have a program that uses resize-able arrays.  I already over
> > -provision the arrays and use slices, but every now and then the data
> > outgrows that array and it needs to be resized.
> >
> > Now, I would like to have these arrays shared between processes
> > spawned via multiprocessing (for fast interprocess communication
> > purposes, not for parallelizing work on an array).  I don't care
> > about mapping to a file on disk, and I don't want disk I/O happening.
> >   I don't care (really) about data being copied in memory on resize.
> > I *do* want the array to be resized "in place", so that the child
> > processes can still access the arrays from the object they were
> > initialized with.
> >
> >
> > I can share arrays easily using arrays that are backed by memmap.
> > Ie:
> >
> >     ```
> >     #Source: http://github.com/rainwoodman/sharedmem
> >
> >
> >     class anonymousmemmap(numpy.memmap):
> >         def __new__(subtype, shape, dtype=numpy.uint8, order='C'):
> >
> >             descr = numpy.dtype(dtype)
> >             _dbytes = descr.itemsize
> >
> >             shape = numpy.atleast_1d(shape)
> >             size = 1
> >             for k in shape:
> >                 size *= k
> >
> >             bytes = int(size*_dbytes)
> >
> >             if bytes > 0:
> >                 mm = mmap.mmap(-1,bytes)
> >             else:
> >                 mm = numpy.empty(0, dtype=descr)
> >             self = numpy.ndarray.__new__(subtype, shape, dtype=descr,
> > buffer=mm, order=order)
> >             self._mmap = mm
> >             return self
> >
> >         def __array_wrap__(self, outarr, context=None):
> >             return
> > numpy.ndarray.__array_wrap__(self.view(numpy.ndarray), outarr,
> > context)
> >     ```
> >
> > This cannot be resized because it does not own it's own data
> > (ValueError: cannot resize this array: it does not own its data).
> > (numpy.memmap has this same issue [0], even if I set refcheck to
> > False and even though the docs say otherwise [1]).
> >
> > arr._mmap.resize(x) fails because it is annonymous (error: [Errno 9]
> > Bad file descriptor).  If I create a file and use that fileno to
> > create the memmap, then I can resize `arr._mmap` but the array itself
> > is not resized.
> >
> > Is there a way to accomplish what I want?  Or, do I just need to
> > figure out a way to communicate new arrays to the child processes?
> >
>
> I guess the answer is no, but the first question should be whether you
> can create a new array viewing the same data that is just larger? Since
> you have the mmap, that would be creating a new view into it.
>
> I.e. your "array" would be the memmap, and to use it, you always rewrap
> it into a new numpy array.
>
> Other then that, you would have to mess with the internal ndarray
> structure, since these kind of operations appear rather unsafe.
>
> - Sebastian
>
>
> > Thanks,
> >   Elliot
> >
> > [0] https://github.com/numpy/numpy/issues/4198.
> >
> > [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.
> > resize.html
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160207/070214d2/attachment.html>