Python does not take up available physical memory

[Tim Delaney]

On 22 October 2012 01:14, Pradipto Banerjee <
pradipto.banerjee at adainvestments.com> wrote:

> I tried this on a different PC with 12 GB RAM. As expected, this time,
> reading the data was no issue. I noticed that for large files, Python takes
> up 2.5x size in memory compared to size on disk, for the case when each
> line in the file is retained as a string within a Python list. As an
> anecdote, for MATLAB, the similar overhead is 2x, slightly lower than
> Python, and each line in the file was retained as string within a MATLAB
> cell. I'm curious, has any one compared the overhead of data in memory for
> other languages like for instance Ruby?


What version of Python were you using? 2.7? 3.2? 3.3?

If you can, try running the same program in Python 3.3 and compare the
amount of memory used and report it here. It sounds like this might be a
case that would greatly benefit from the new string representation in 3.3.

If you're using Python 3.x then the "byte" and "bytearray" types might be
of interest to you:
http://docs.python.org/py3k/library/stdtypes.html#binary-sequence-types-bytes-bytearray-memoryview

Alternatively, the "array" type might be useful:
http://docs.python.org/py3k/library/array.html

As to the core problem, I can only echo what others have said - only hold
in memory what you absolutely have to. There are various techniques to
avoid holding unnecessary data in memory that have been mentioned. One I
haven't seen here yet (I may have missed it) is dumping the data into a
database of some form and using it's capabilities.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20121022/46cc6dff/attachment.html>