using mmap on large (> 2 Gig) files

[Paul Rubin]

"sturlamolden" <sturlamolden at yahoo.no> writes:
> However, "memory mapping" a file by means of fseek() is probably more
> efficient than using UNIX' mmap() or Windows'
> CreateFileMapping()/MapViewOfFile().

Why on would you think that?!  It is counterintuitive.  fseek beyond
whatever is buffered in stdio (usually no more than 1kbyte or so)
requires a system call, while mmap is just a memory access.

> In Python, we don't always need the file memory mapped, we normally
> just want to use slicing-operators, for-loops and other goodies on
> the file object -- i.e. we just want to treat the file as a Python
> container object. There are many ways of achieving that.

Some of the time we want to share the region with other processes.
Sometimes we just want random access to a big file on disk without
having to do a lot of context switches seeking around in the file.

> There are in any case room for improving Python's mmap object.

IMO it should have some kind of IPC locking mechanism added, in
addition to the offset stuff suggested.