Python does not take up available physical memory

[Steven D'Aprano]

On Fri, 19 Oct 2012 14:03:37 -0500, Pradipto Banerjee wrote:

> Thanks, I tried that. Still got MemoryError, but at least this time
> python tried to use the physical memory. What I noticed is that before
> it gave me the error it used up to 1.5GB (of the 2.23 GB originally
> showed as available) - so in general, python takes up more memory than
> the size of the file itself.

Well of course it does. Once you read the data into memory, it has its 
own overhead for the object structure.

You haven't told us what the file is or how you are reading it. I'm going 
to assume it is ASCII text and you are using Python 2.

py> open("test file", "w").write("abcde")
py> os.stat("test file").st_size
5L
py> text = open("test file", "r").read()
py> len(text)
5
py> sys.getsizeof(text)
26

So that confirms that a five byte ASCII string takes up five bytes on 
disk but 26 bytes in memory as an object.

That overhead will depend on what sort of object, whether Unicode or not, 
the version of Python, and how you read the data.

In general, if you have a huge amount of data to work with, you should 
try to work with it one line at a time:

for line in open("some file"):
    process(line)


rather than reading the whole file into memory at once:

lines = open("some file").readlines()
for line in lines:
    process(line)



-- 
Steven