Total maximal size of data

Alexander Moibenko moibenko at fnal.gov
Mon Jan 25 14:05:28 EST 2010


I have a simple question to which I could not find an answer.
What is the total maximal size of list including size of its elements?
I do not like to look into python source.
Here is a code example:
import struct
KB=1024
MB=KB*KB
GB=MB*KB
buf=[]
bs=32*KB
n=4*GB/bs
print "N",n
i=0
size=0L
while i < n:
    data = struct.pack("%ss" % (bs,), "")
    buf.append(data)
    size = size+bs
    if size % (100*MB) == 0:
        print "SIZE", size/MB, "MB"
    i=i+1
while 1:
    # to keep script running while I am looking at the machine status
    pass

Here is what I get on 32-bit architecture:
cat /proc/meminfo
MemTotal:      8309860 kB
MemFree:       5964888 kB
Buffers:         84396 kB
Cached:         865644 kB
SwapCached:          0 kB
......
The program output:
N 131072
SIZE 100 MB
SIZE 200 MB
SIZE 300 MB
SIZE 400 MB
SIZE 500 MB
SIZE 600 MB
SIZE 700 MB
SIZE 800 MB
SIZE 900 MB
SIZE 1000 MB
SIZE 1100 MB
SIZE 1200 MB
SIZE 1300 MB
SIZE 1400 MB
SIZE 1500 MB
SIZE 1600 MB
SIZE 1700 MB
SIZE 1800 MB
SIZE 1900 MB
SIZE 2000 MB
SIZE 2100 MB
SIZE 2200 MB
SIZE 2300 MB
SIZE 2400 MB
SIZE 2500 MB
SIZE 2600 MB
SIZE 2700 MB
SIZE 2800 MB
SIZE 2900 MB
SIZE 3000 MB
Traceback (most recent call last):
  File "bs.py", line 14, in ?
    data = struct.pack("%ss" % (bs,), "")
MemoryError
++++++++++++++++++++
The number of list elements for a given block size is 131072.
If I change block size the script traces back at the same total size 3000MB.
Somewhere I read that list could have
2147483647 items, on most platforms.
Somewhere else that it is
*536,870,912 
(http://stackoverflow.com/questions/855191/how-big-can-a-python-array-get)**
*But what is the maximal size of the whole list including the size of 
its elements?
Thanks.




More information about the Python-list mailing list