[SciPy-user] Memory usage of scipy.io.loadmat

[Ariel Rokem]

Hi everyone,

I have stumbled on some interesting behavior of scipy.io.loadmat. The short
of it - it looks like loadmat is gobbling up memory in some unjustified
manner and releasing it under some strange circumstances.

Here's the long story. This is all happening on a Mac OS 10.5.7, running EPD
4.0.30001 (Python 2.5.2), but with a relatively new version of scipy (see
below).

I start ipython -pylab in one terminal and run 'top' in another, in order to
monitor the memory usage. Here's what I get initially:

PhysMem:  417M wired,  449M active,  183M inactive, 1055M used, 3041M free.

Then:

In [1]: import scipy

In [2]: scipy.__version__
Out[2]: '0.8.0.dev5606'

In [3]: import scipy.io as sio

Here's what it looks like now:

PhysMem:  418M wired,  450M active,  183M inactive, 1058M used, 3038M free.

So far, so good. I read in a large matfile with tons of data in it:

In [4]: a =
sio.loadmat('/Users/arokem/Projects/SchizoSpread/Scans/SMR033109_MC/Gray/Original/TSeries/Scan1/tSeries1.mat')

PhysMem:  419M wired, 1024M active,  183M inactive, 1632M used, 2464M free.

So - about 600 MB of memory is taken up by this new variable.

Now to the wierdness:

In [5]: b
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/Users/arokem/<ipython console> in <module>()

NameError: name 'b' is not defined

Of course - 'b' doesn't exist!

But now, the memory usage has dramatically gone down:

PhysMem:  420M wired,  740M active,  183M inactive, 1350M used, 2746M free.

So - just invoking an error in the ipython command line has freed up 300 MB.
Where did they come from? I tried different things - assigning other
variables doesn't seem to free up this memory. Neither do calls to other
functions. Except "plot()", which does seem to do the trick for some reason.
Interestingly, when I run all this in a python interactive session (and not
ipython), I get a similar memory usage initially. Calling a non-existent
variable does not free up the memory, but other things do. For example,
import matplotlib.pylab into the namespace did the trick. Does anyone have
any idea what is going on?

Thanks --

Ariel


-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20090630/3865a477/attachment.html>