Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 18:42, Chris Angelico wrote:

> Are you able to share the program? I could try it on my system and see
> if the same thing happens.

Yes, it is on GitHub (use the fixes branch): 
https://github.com/ctn-archive/gosmann-frontiers2017/tree/fixes
Installation instructions are in the readme.
The command I'm running is `python scripts/log_reduction.py spaun`

I was looking at the output of htop to see the memory consumption. The 
`time` built-in of the zsh might report a lower memory usage (I'm 
running it right now to make sure), which would also be surprising. But 
I'm certain that Python 3.6 uses more memory because the run time 
increases due to using the swap partition.

If you run the program, it will take an initial 10–15 minutes of 
processing during which the memory usage should increase to roughly 
10GB. Then the output “[INFO] Optimizing model...” will be printed 
after which the memory usage continues to increase to around 30GB (maybe 
a bit more with Python 3.6). This constitutes a first “optimization 
pass” that will be ended with the message “[INFO] Pass 1 [views]: 
Reduced 996917 to 913666 operators in 62.519006s.” Then in the second 
optimization pass the memory consumption will continue to increase with 
Python 3.6 (up to ~60GB being consumed, where less than 5GB should be 
due to other processes, though the virtual and resident memory reported 
for the Python process are only ~42GB and ~30GB respectively) and in 
later passes it will start decreasing again. With Python 3.5 the memory 
consumption stays about 30GB and decreases in later passes.

Let me know if you have more questions.