memory control in Python

[Terry Reedy]

On 8/15/2015 3:21 AM, dieter wrote:
> Ping Liu <yanzhipingliu at gmail.com> writes:
>> ...
>> For small cases, Python works well. But if we consider longer time period.
>> then it would fail due to the memory usage issues. We have tested several
>> case studies to check the memory use for different time period, including
>> 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
>> each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
>> are feasible while the last case gives out the memory error.
>>
>> When we are testing the forth case, the memory error comes out while
>> creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
>> Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
>> 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
>>
>> The solver is CPLEX (academic). It turns out that the solver is taking a
>> lot of memory as you can see in the memory test report. for the first three
>> cases, different memory usage is observed, and it grows up dramatically
>> with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
>> 19.5 MB; 3) solver memory usage: 830.0742 MB.

Make sure that the solver is using numpy arrays.

> The C implementation of Python (called "CPython") does not use
> memory compaction and places most of its objects on the heap.
> Those implementations tend to suffer from memory fragmentation in
> long running processes with large memory use.

This can be helped somewhat by reusing arrays instead of deleting and 
creating a new one.


> You may consider "Jython" (a Python implementation in "Java") or
> "IronPython" (a ".Net" based Python implementation) to get a
> Python implementation with memory compaction.
>


-- 
Terry Jan Reedy