[Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB

[Citi, Luca]

Hello!
I have access to both a 32bit and a 64bit linux machine.

I had to change your code (appended) because I got an error about
not being able to create a mmap larger than the file.
Here are the results...

On the 32bit machine:

lciti at xps2:~$ python /tmp/ppp.py
Created 1 writeable mmaps containing 100 MB
Created 2 writeable mmaps containing 200 MB
Created 3 writeable mmaps containing 300 MB
Created 4 writeable mmaps containing 400 MB
Created 5 writeable mmaps containing 500 MB
[......]
Created 24 writeable mmaps containing 2400 MB
Created 25 writeable mmaps containing 2500 MB
Created 26 writeable mmaps containing 2600 MB
Created 27 writeable mmaps containing 2700 MB
Created 28 writeable mmaps containing 2800 MB
Created 29 writeable mmaps containing 2900 MB
Created 30 writeable mmaps containing 3000 MB
Removing mmaps...
Done...
Traceback (most recent call last):
  File "/tmp/ppp.py", line 19, in <module>
    mm = mmap.mmap(f.fileno(), 0)
mmap.error: [Errno 12] Cannot allocate memory




On the 64bit machine I can create 510 mmaps
both with bytes_per_mmap at 100MiB and 1GiB:

Created 1 writeable mmaps containing 1000 MB
Created 2 writeable mmaps containing 2000 MB
Created 3 writeable mmaps containing 3000 MB
Created 4 writeable mmaps containing 4000 MB
Created 5 writeable mmaps containing 5000 MB
Created 6 writeable mmaps containing 6000 MB
[......]
Created 501 writeable mmaps containing 501000 MB
Created 502 writeable mmaps containing 502000 MB
Created 503 writeable mmaps containing 503000 MB
Created 504 writeable mmaps containing 504000 MB
Created 505 writeable mmaps containing 505000 MB
Created 506 writeable mmaps containing 506000 MB
Created 507 writeable mmaps containing 507000 MB
Created 508 writeable mmaps containing 508000 MB
Created 509 writeable mmaps containing 509000 MB
Created 510 writeable mmaps containing 510000 MB
Removing mmaps...
Done...
Traceback (most recent call last):
  File "/tmp/ppp.py", line 19, in <module>
    mm = mmap.mmap(f.fileno(), 0)
mmap.error: [Errno 24] Too many open files



I do not even have 510GiB free in the disk. But I
think that is because the ext3 filesystem allows
sparse files.

I think this shows that the maximum mapped space
cannot be more than the maximum address space
which is 2**64 for 64bit machines, 2GiB for windows32
and 3GiB for linux32.

Under WindowsXP, you can try to increase it from 2GiB to
3GiB using the /3GB switch in the boot.ini

Best,
Luca



### CODE ###

import itertools
import mmap
import os

files = []
mmaps = []
file_names= []
mmap_cap=0
bytes_per_mmap = 100 * 1024 ** 2
try:
    for i in itertools.count(1):
        file_name = "/home/lciti/%d.tst" % i
        file_names.append(file_name)
        f = open(file_name, "w+b")
        files.append(f)
        f.seek(bytes_per_mmap)
        f.write('a')
        f.seek(0)
        mm = mmap.mmap(f.fileno(), 0)
        mmaps.append(mm)
        mmap_cap += bytes_per_mmap
        print "Created %d writeable mmaps containing %d MB" % (i,mmap_cap/(1024**2))

#Clean up
finally:
    print "Removing mmaps..."
    for mm, f, file_name in zip(mmaps, files, file_names):
        mm.close()
        f.close()
        os.remove(file_name)
    print "Done..."