using mmap on large (> 2 Gig) files

[Tim Roberts]

"sturlamolden" <sturlamolden at yahoo.no> wrote:
>
>However, "memory mapping" a file by means of fseek() is probably more
>efficient than using UNIX' mmap() or Windows'
>CreateFileMapping()/MapViewOfFile().

My goodness, do I disagree with that!  At least on Windows, I/O on a file
mapped with MapViewOfFile uses the virtual memory pager -- the same
mechanism used by the swap file.  Because it is so heavily used, that is
some of the most well-optimized code in the system.

>We can implement a container object backed by a binary file just as
>efficient (and possibly even more efficient) without using the OS'
>memory mapping facilities. The major advantage is that we can
>"pseudo-memory map" a lot more than a 32 bit address space can harbour.

Both the Unix mmap and the Win32 MapViewOfFile allow a starting byte
offset.  It wouldn't be rocket science to extend Python's mmap to allow
that.

>There are in any case room for improving Python's mmap object.

Here we agree.
-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.