how to python to use virtual memory?

[Joonas Liik]

On 26 June 2016 at 04:47, Ho Yeung Lee <davidbenny2000 at gmail.com> wrote:
> what is the command or code to write to use virtual memory if i use extra
> 20 GB from hard disk as memory, means from 70GB memory to 90GB memory
> and left 10GB for file?
>
> Michael Torrie於 2016年6月25日星期六 UTC+8上午11時00分36秒寫道：
>> On 06/24/2016 08:44 PM, Dennis Lee Bieber wrote:
>> >     I don't know how Linux handles swap disk -- Windows normally sets the
>> > swap space to ~2X physical memory (for small RAM -- my 12GB system has a
>> > 12GB swap and suggests 18GB).
>>
>> Linux typically uses a user-set swap partition.  The old rule of thumb
>> was to make the swap partition 2x the size of RAM. Now, though, for most
>> installations with lots of RAM, 1:1 is often used.
>>
>> However, if the OP's program really requires 70 to 100 GB of space,
>> relying on the virtual memory system to do this (64-bit only of course)
>> is a mistake.  The system will simply thrash itself to death on any OS
>> at that level of over-commit.  If he has that much data, he needs to
>> employ techniques for working with the data directly on disk himself.  I
>> highly doubt these big data sets that large companies work rely simply
>> on the OS to manage it!
>
> --
> https://mail.python.org/mailman/listinfo/python-list

You really should avoid relying on the OS trick of pretending you have
more ram than you actually do. this can easily cause several
magnitudes worth of slowdown. if you succeed i bet the next post to
this list will be titled "why is my program taking forever to run". in
stead try to partition the work in to chunks and only keep the truly
necessary working set loaded at a time.

For example if you load data from an xml or json file it will likely
include more data than you need to do your calculations. perhaps it is
possible to throw some of it away or at least remove it for the time
of the memory intensive step.

there are ways to allow your program to use more memory ofc. but this
may run in to OS limits (like the aforementioned 32 bit windows issue)
and will likely incur heavy performance penalties. you should avoid it
if at all possible.
even if you do you will likely benefit from adapting your algorithm.