Fast forward-backward (write-read)

[Virgil Stokes]

On 24-Oct-2012 02:06, Oscar Benjamin wrote:
> On 23 October 2012 15:31, Virgil Stokes <vs at it.uu.se> wrote:
>> I am working with some rather large data files (>100GB) that contain time
>> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
>> format. I perform various types of processing on these data (e.g. moving
>> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
>> manner and only a small number of these data need be stored in RAM when
>> being processed. When performing Kalman-filtering (forward in time pass, k =
>> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
>> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
>> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
>> variables saved to an external file from the forward pass, in reverse order
>> --- from last written to first written.
>>
>> Finally, to my question --- What is a fast way to write these variables to
>> an external file and then read them in backwards?
> You mentioned elsewhere that you are using numpy. I'll assume that the
> data you want to read/write are numpy arrays.
>
> Numpy arrays can be written very efficiently in binary form using
> tofile/fromfile:
>
>>>> import numpy
>>>> a = numpy.array([1, 2, 5], numpy.int64)
>>>> a
> array([1, 2, 5])
>>>> with open('data.bin', 'wb') as f:
> ...   a.tofile(f)
> ...
>
> You can then reload the array with:
>
>>>> with open('data.bin', 'rb') as f:
> ...   a2 = numpy.fromfile(f, numpy.int64)
> ...
>>>> a2
> array([1, 2, 5])
>
> Numpy arrays can be reversed before writing or after reading using;
>
>>>> a2
> array([1, 2, 5])
>>>> a2[::-1]
> array([5, 2, 1])
>
> Assuming you wrote the file forwards you can make an iterator to yield
> the file in chunks backwards like so (untested):
>
> def read_backwards(f, dtype, chunksize=1024 ** 2):
>      dtype = numpy.dtype(dtype)
>      nbytes = chunksize * dtype.itemsize
>      f.seek(0, 2)
>      fpos = f.tell()
>      while fpos > nbytes:
>          f.seek(fpos, 0)
>          yield numpy.fromfile(f, dtype, chunksize)[::-1]
>          fpos -= nbytes
>      yield numpy.fromfile(f, dtype)[::-1]
>
>
> Oscar
Ok Oscar,
Thanks for the tip and I will look into this more.