Fast forward-backward (write-read)

[Tim Chase]

On 10/23/12 09:31, Virgil Stokes wrote:
> I am working with some rather large data files (>100GB) that contain time series 
> data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform 
> various types of processing on these data (e.g. moving median, moving average, 
> and Kalman-filter, Kalman-smoother) in a sequential manner and only a small 
> number of these data need be stored in RAM when being processed. When performing 
> Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an 
> external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These 
> are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0). 
> Thus, I will need to input these variables saved to an external file from the 
> forward pass, in reverse order --- from last written to first written.
> 
> Finally, to my question --- What is a fast way to write these variables to an 
> external file and then read them in backwards?

Am I missing something, or would the fairly-standard "tac" utility
do the reversal you want?  It should[*] be optimized to handle
on-disk files in a smart manner.

Otherwise, if you can pad the record-lengths so they're all the
same, and you know the total number of records, you can seek to
Total-(RecSize*OneBasedOffset) and write the record,optionally
padding if you need/can.  At least on *nix-like OSes, you can seek
into a sparse-file with no problems (untested on Win32).

-tkc



[*]
Just guessing here. Would be disappointed if it *wasn't*.