[IPython-dev] ipython memory usage problem

[cadourian]

I could not provide an example that causes the problem cause the setup to run
it would be too complicated however I think we should be able to make
progress.

I'll explain. I ran two examples. Both examples have the same pseudocode
template

-------------
Procedure A: 
-------------
a) Clear the memory on the engines using the following commands

    securityFile =
r'%s\EngineConfig\profile_default\security\ipcontroller-client.json' %
projectRoot
    rc = Client(securityFile)
    idSubset = rc.ids

    dview = rc[idSubset]
    dview.execute('%reset')
    dview.execute('import gc')
    dview.execute('gc.collect()')


b) Measure memory usage on each cluster engine
c) call a function x using dview.map_sync(x, ....)
d) Measure memory usage on each cluster engine

#1) memory usage on engines does not increase when Procedure A is called
multiple times. In this case, the function x used in dview.map_sync(x, ....)
allocates an array with numpy and returns it

#2) In this case, memory increases when Procedure A is called multiple
times. The difference is that function x used in dview.map_sync(x, ....)
loads DLLs and executes loaded functions.

What I don't understand is that I'm calling a function (x) which is
obviously causing the problems. But, I would have expected that when the
function call returned, any objects created within would be destroyed.
Moreover, I purposefully called garbage collect to eliminate any remaining
objects before measuring memory usage.

The question is whether IPython handles functions a little differently than
Python? Any suggestions appreciated. 





--
View this message in context: http://python.6.x6.nabble.com/ipython-memory-usage-problem-tp5174336p5174675.html
Sent from the IPython - Development mailing list archive at Nabble.com.