[SciPy-dev] Memory mapped files in scipy core

[Travis Oliphant]

Joe Harrington wrote:

>I'm not familiar with the mmap interface, but these insertion tricks
>sound like they solve a particularly unpleasant problem of IDL that I
>hit a lot.
>
>It's considered nice in CS to allocate space on the fly, since it
>keeps your allocation with the code that uses it.  It's particularly
>useful if you have an unknown amount of data coming in.  To that end,
>I have a routine, concat, that allows you to tack on an array to any
>side of an existing array:
>
>x = concat(2, x, y)
>
>puts x next to y in the second dimension.  The arrays may have any
>dimension and shape.  If they're different shapes, concat fills any
>void space in with a pad value.  If I'm reading in a dataset of some
>thousands of images, I can just put that in a loop and then ask the
>final array how big it is, rather than "peeking" at some ancillary
>data to find out how much space to pre-allocate in x.  The
>pre-allocation line (which has to be outside the loop in IDL, or has
>to be protected by an if) is unnecessary.
>  
>
This looks like a good routine to put in scipy_core if you don't mind 
contributing it.

I can see that resizing memory mapped arrays and inserting into them 
would be useful.  I'm not familiar enough with the memory-mapped-file to 
understand how inserting into a memory-mapped file would work. 

It looks like numarray's implementation actually copies data into 
RAM-based buffers on insertion and re-sizing which would seem to negate 
the benefits you speak of.

-Travis