Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 20:12, Chris Angelico wrote:

> On Tue, Mar 28, 2017 at 11:00 AM, Chris Angelico <rosuav at gmail.com> 
> wrote:
>> In any case, I've installed nvidia-opencl-dev and it seems to be
>> happy. How long should I expect this to run for?
>>
>> Now testing under CPython 3.7.
>>
>
> It ran for about 14 minutes, then memory usage spiked and went into
> the page file. Used roughly 8GB of RAM prior to that point, I think?
> Unfortunately, I don't think my hard disk is fast enough for this to
> run through the page file in any reasonable time.

Yeah, it will be problematic with less than 32GB.

My latest findings: The zsh `time` built-in reports a maximum of only 
30GB used which seems to correspond to the resident memory size, while 
the virtual memory size exceeds 40GB for the process and the total 
memory allocated over all processes exceeds 60GB (with the main 
contribution from the process that supposedly only uses ~40GB).

My best idea about what's going on at the moment is that memory 
fragmentation is worse in Python 3.6 for some reason. The virtual memory 
size indicates that a large address space is acquired, but the resident 
memory size is smaller indicating that not all of that address space is 
actually used. In fact, the code might be especially bad to 
fragmentation because it takes a lot of small NumPy arrays and 
concatenates them into larger arrays. But I'm still surprised that this 
is only a problem with Python 3.6 (if this hypothesis is correct).

Jan