memory control in Python

[dieter]

Ping Liu <yanzhipingliu at gmail.com> writes:
> ...
> For small cases, Python works well. But if we consider longer time period.
> then it would fail due to the memory usage issues. We have tested several
> case studies to check the memory use for different time period, including
> 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
> each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
> are feasible while the last case gives out the memory error.
>
> When we are testing the forth case, the memory error comes out while
> creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
> Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
> 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
>
> The solver is CPLEX (academic). It turns out that the solver is taking a
> lot of memory as you can see in the memory test report. for the first three
> cases, different memory usage is observed, and it grows up dramatically
> with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
> 19.5 MB; 3) solver memory usage: 830.0742 MB.

The C implementation of Python (called "CPython") does not use
memory compaction and places most of its objects on the heap.
Those implementations tend to suffer from memory fragmentation in
long running processes with large memory use.

You may consider "Jython" (a Python implementation in "Java") or
"IronPython" (a ".Net" based Python implementation) to get a
Python implementation with memory compaction.