[Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB

[David Cournapeau]

Kim Hansen wrote:
> 2009/7/24 Citi, Luca <lciti at essex.ac.uk>:
>   
>> Hello!
>> I have access to both a 32bit and a 64bit linux machine.
>>
>> I had to change your code (appended) because I got an error about
>> not being able to create a mmap larger than the file.
>> Here are the results...
>>
>> On the 32bit machine:
>>
>> lciti at xps2:~$ python /tmp/ppp.py
>> Created 1 writeable mmaps containing 100 MB
>> Created 2 writeable mmaps containing 200 MB
>> Created 3 writeable mmaps containing 300 MB
>> Created 4 writeable mmaps containing 400 MB
>> Created 5 writeable mmaps containing 500 MB
>> [......]
>> Created 24 writeable mmaps containing 2400 MB
>> Created 25 writeable mmaps containing 2500 MB
>> Created 26 writeable mmaps containing 2600 MB
>> Created 27 writeable mmaps containing 2700 MB
>> Created 28 writeable mmaps containing 2800 MB
>> Created 29 writeable mmaps containing 2900 MB
>> Created 30 writeable mmaps containing 3000 MB
>> Removing mmaps...
>> Done...
>> Traceback (most recent call last):
>>  File "/tmp/ppp.py", line 19, in <module>
>>    mm = mmap.mmap(f.fileno(), 0)
>> mmap.error: [Errno 12] Cannot allocate memory
>>
>>
>>
>>
>> On the 64bit machine I can create 510 mmaps
>> both with bytes_per_mmap at 100MiB and 1GiB:
>>
>> Created 1 writeable mmaps containing 1000 MB
>> Created 2 writeable mmaps containing 2000 MB
>> Created 3 writeable mmaps containing 3000 MB
>> Created 4 writeable mmaps containing 4000 MB
>> Created 5 writeable mmaps containing 5000 MB
>> Created 6 writeable mmaps containing 6000 MB
>> [......]
>> Created 501 writeable mmaps containing 501000 MB
>> Created 502 writeable mmaps containing 502000 MB
>> Created 503 writeable mmaps containing 503000 MB
>> Created 504 writeable mmaps containing 504000 MB
>> Created 505 writeable mmaps containing 505000 MB
>> Created 506 writeable mmaps containing 506000 MB
>> Created 507 writeable mmaps containing 507000 MB
>> Created 508 writeable mmaps containing 508000 MB
>> Created 509 writeable mmaps containing 509000 MB
>> Created 510 writeable mmaps containing 510000 MB
>> Removing mmaps...
>> Done...
>> Traceback (most recent call last):
>>  File "/tmp/ppp.py", line 19, in <module>
>>    mm = mmap.mmap(f.fileno(), 0)
>> mmap.error: [Errno 24] Too many open files
>>
>>
>>
>> I do not even have 510GiB free in the disk. But I
>> think that is because the ext3 filesystem allows
>> sparse files.
>>
>> I think this shows that the maximum mapped space
>> cannot be more than the maximum address space
>> which is 2**64 for 64bit machines, 2GiB for windows32
>> and 3GiB for linux32.
>>
>> Under WindowsXP, you can try to increase it from 2GiB to
>> 3GiB using the /3GB switch in the boot.ini
>>
>> Best,
>> Luca
>>
>>     
> Hi Luca, thx for trying. Your test clearly shows that 32 bits imposes
> a severe limitation.
>
> I tried adding the /3GB switch to boot.ini as you suggested:
> multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP
> Professional" /noexecute=optin /fastdetect /3GB
> and rebooted the system.
>
> Unfortunately that did not change anything for me. I still hit a hard
> deck around 1.9 GB. Strange.
>   

The 3Gb thing only works for application specifically compiled for it:

http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx

I somewhat doubt python is built with this, but you could check this in
python sources to be sure,

cheers,

David