[Numpy-discussion] Managing Rolling Data

[Alexander Michael]

I'm new to numpy and looking for advice on setting up and managing
array data for my particular problem. I'm collecting observations of P
properties for N objects over a rolling horizon of H sample times. I
could conceptually store the data in three-dimensional array with
shape (N,P,H) that would allow me to easily (and efficiently with
strided slices) compute the statistics over both N and H that I am
interested in. This is great, but the rub is that H, an interval of T,
 is a rolling horizon. T is to large to fit in memory, so I need to
load up H, perform my calculations, pop the oldest N x P slice and
push the newest N x P slice into the data cube. What's the best way to
do this that will maintain fast computations along the one-dimensional
slices over N and H? Is there a commonly accepted idiom?

Fundamentally, I see two solutions. The first would be to essentially
perform a memcpy to propagate the data. The second would be to manage
the N x P slices as H discontiguous memory blocks and merely reorder
the pointers with each new sample. Can I do either of these with
numpy?

Thanks,
Alex