Python does not take up available physical memory

[Chris Kaynor]

On Fri, Oct 19, 2012 at 12:03 PM, Pradipto Banerjee
<pradipto.banerjee at adainvestments.com> wrote:
>
> Thanks, I tried that. Still got MemoryError, but at least this time python tried to use the physical memory. What I noticed is that before it gave me the error it used up to 1.5GB (of the 2.23 GB originally showed as available) - so in general, python takes up more memory than the size of the file itself.
>

By default, 32-bit processes under Windows are restricted to only
using a maximum of 2GB of user-space memory, with the other 2GB
reserved for kernel usage. There are settings for Windows and the
application when compiled to up the limit to 3GB.

You can follow the MSDN entry at
http://msdn.microsoft.com/en-us/library/windows/desktop/bb613473(v=vs.85).aspx
to enable 3GB of space on your Windows box (it requires a reboot),
however I believe the standard Python installations do not have
IMAGE_FILE_LARGE_ADDRESS_AWARE set, so you will also have to compile a
custom Python instance to get Python to be able to use 3GB of space.
Note that doing so can result in instability and other side-effects:
the Windows options can reduce performance in some cases, and the
process setting can cause crashes or other errors if the application
presumes that user memory always starts with a 0-bit, often used to
save memory in older software.