mmap caching

["Martin v. Löwis"]

Laszlo Nagy schrieb:
> 
>> In fact, memory that is read in because of mmap should *never* cause
>> a MemoryError. 
> This is certainly not true. You can run out of virtual address space by
> reading data from a memory mapped file.

That is true, but not what I said. I said you cannot run out of memory
*while reading it*. You can only run out of virtual address space when
you invoke mmap.mmap itself (and when the application later tries to
allocate more virtual address space through VirtualAlloc).

>> Python calls MapViewOfFile when mmap.mmap is invoked,
>> at which point the operating commits to providing that much address
>> space to the application, along with backing storage on disk
>> (typically, from the file being mapped, unless it is an anonymous
>> map). Later access to the mapped range cannot fail (except for
>> hardware errors), and if it would, you wouldn't see a MemoryError.
>>   
> Hmm, maybe I'm wrong. Are you sure that Windows allocates the size of
> the whole file in terms of memory address space?

Yes, I am. See MapViewOfFile, at

http://msdn2.microsoft.com/en-us/library/aa366761.aspx

"Mapping a file makes the specified portion of a file visible in the
address space of the calling process."

Notice allocating address space doesn't consume much memory (it
consumes a little memory for the page tables).

>  I also wrote a program
> before (in Delphi). That program was playing a memory mapped wave file.
> From the task manager, I have seen that "used memory" was growing as the
> program was playing the wave file. For me, this indicates that Windows
> extends the mapped address space in chunks.

You are misinterpreting the data. I'm not sure what precisely
"used memory" is, most likely it is the working set of the process, i.e.
the amount the number of physical pages that are allocated for the
process. That is typically much smaller than the address space, since
many pages will be paged out (or not yet read in at all).

You need to display the virtual address space in the task manager
to determine how much address space the application is using.

Regards,
Martin