pylab/matplotlib large plot memory management - bug? or tuning parameter needed?

[Carl Banks]

On Oct 19, 12:15 pm, bdb112 <boyd.blackw... at gmail.com> wrote:
> Summary:
>
> It is not straightforward to avoid memory leaks/consumption in pylab.
> If we define
> x = arange(1e6)     # adjust size to make the increment visible, yet
> fast enough to plot
> # then repetition of
> plot(x,hold=0)   # consumes increasing memory according to ubuntu
> system monitor
>
> Details:
> #versions: ubuntu 9.04 standard (py 2.6.2, ipython 0.91, matplotlib
> 0.98.5.2) , (results here)
> #or win32 matplotlib '0.98.3', python 2.5 (similar, although mem usage
> limits and reuses after 3-4 plot()s
> First, turn off output caching in ipython because it consumes memory
> (as it should)
>
> ipython -pylab -cs 0
>
> plot(x,hold=0)
> plot(x,hold=0)
> plot(x,hold=0)
>
> # closing the window doesn't help much, neither does close() or any of
> the below individually
> # antidote
> plot(hold=0); gcf().clf(); close()  # first often doesn't help!
> plot(hold=0); gcf().clf(); close()  # need all three!
> plot(hold=0); gcf().clf(); close()
>
> As stated above, the windows version apparently starts more aggressive
> garbage collection after 3-4 plots.
> The ubuntu version reaches my system memory limit (1GB) without
> reclaiming memory - i.e. memory usage just keeps growing until swap
> space is used when array is 2e6 elements, consuming 100mB per plot.
> For 1e6 elements, memory usage grows for about 10 50MB steps, and then
> some garbage collection seems to happen, alothough more can be freeed
> with the triple line above.
>
> 1/ I am running under VMWare so maybe VMWare isn;t reporting the
> correct physical memory size to ubuntu/python - how to check this?
>
> 2/ possible bug - why doesn't closing the plot window release all
> memory it uses?  Especially when this approaches machine memory size.
>
> 3/ Are there python/matplotlib memory management tuning parameters I
> can tweak?


First off, let's clear up a couple misconceptions.

1. CPython garbage collection is not triggered by physical or OS
memory constraints.  Objects are garbage collected in CPython for one
reason: their reference count goes to zero.  Python on Windows doesn't
"start more aggressive garbage collection", since there is no
aggressive garbage collection.  (There is such thing aggressive cycle
detection; however it's also not triggered by physical memory. I doubt
cycles are the issue for you.)

2. References to objects can be kept around for various reasons, thus
keeping their reference counts above zero.  So even if you del the
variable the object was bound to, there might be another reference
somewhere, in which case the object won't be garbage collected right
away.  When you closed the Pylab window, and memory wasn't freed right
away, doesn't mean it's a bug in Python, or even Pylab.

Somehow on Ubuntu, pylab is either keeping references to old data
around (what I like to call a memory clog, as opposed to a memory
leak), or is allocating memory in such a way that it can't unwind the
heap and reclaim memory from a large array that was freed.  The latter
seems unlikely to me, because any large memory allocations are
probably mmaped under the covers(which means it gets its own page and
the OS can move it around).

If it is a bug, it's probably with Pylab, but you never know.


Carl Banks