[Numpy-discussion] Managing Rolling Data

[Anne Archibald]

On 21/02/07, Alexander Michael <lxander.m at gmail.com> wrote:
> On 2/21/07, Mike Ressler <mike.ressler at alum.mit.edu> wrote:
> > Would loading your data via memmap, then slicing it, do your job
> > (using numpy.memmap)? ...
>
> Interesting idea. I think Anne's suggestion that sliced assignment
> will reduce to an efficient memcpy fits my needs a bit better than
> memmap because I'll be pushing new N x P samples into the array that
> will be arriving while the monitor is running.

If you want a record of all of them on disk anyway, with careful
management you can read from a file as it's being written, though
you'll need some rather exotic numpy hackery to have an ever-growing
array. It might actually be nice to have a little wrapper object for
this sort of thing, as you can't be the only one who needs it.

> Actually, I'm hoping sliced self-assignment is as efficient as memmove
> (i.e. without creating temporaries), since the dst and src are
> overlapping, but I haven't tried it yet to confirm if this is
> relatively efficient.

I think it is almost as efficient as memmove; in particular, it
doesn't create any temporaries (be careful which way you do the
assignment, in fact, or you'll have an array of all the same thing)
but it does use a for loop in C (maybe several) instead of a
highly-optimized C library bulk copy. (I'm not sure about this - it
could be special-cased in the assignment code, but I don't think it
is.)

Anne