[SciPy-dev] Memory mapped files in scipy core

[Travis Oliphant]

Paul Barrett wrote:

> On 11/20/05, *Travis Oliphant* <oliphant at ee.byu.edu 
> <mailto:oliphant at ee.byu.edu>> wrote:
>
>
>     On a related, but orthogonal note:
>
>     My understanding is that using memory-mapped files for *very* large
>     files will require modification to the mmap module in Python ---
>     something I think we should push.  One part of that process would
>     be to
>     add the C-struct array interface to the mmap module and the buffer
>     object -- perhaps this is how we get the array interface into Python
>     quickly.   Then, if we could make a base-type mmap that did not
>     use the
>     buffer interface or the sequence interface (similar to the
>     bigndarray in
>     scipy_core) and therefore by-passed the problems with Python in those
>     areas, then the current mmap object could inherit from the base class
>     and provide current functionality while still exposing the array
>     interface for access to >2GB files on 64-bit systems.
>
>
> Is it really necessary to embed the C-struct array interface in mmap?  
> I thought that Python array interface was the standard mechanism for 
> interfacing with array objects.  Or do I misunderstand you.
>
The C-struct interface is faster and a bit easier to code in C.   
Currently the C-struct interface is how numarray/ Numeric  and scipy 
core are all talking to each other. 

One can use either the C-struct interface or the Python interface for 
implementing the array interface.  I suspect we would want both on the 
memory-mapped object.  The most important element of the proposal is 
elimination of the buffer and sequence protocols on a new base-class 
mmap object.  Then users could use this object to memory-map very large 
files.

-Travis