multiprocessing / forking memory usage

[Piet van Oostrum]

>>>>> Randall Smith <randall at tnr.cc> (RS) wrote:

>RS> I'm trying to get a grasp on how memory usage is affected when forking as
>RS> the multiprocessing module does.  I've got a program with a parent process
>RS> using wx and other memory intensive modules. It spawns child processes (by
>RS> forking) that should be very lean (no wx required, etc). Based on
>RS> inspection using "ps v" and psutil, the memory usage (rss) is much higher
>RS> than I would expect for the subprocess.

The child is a clone of the parent. So both its virtual memory usage and
its resident memory usage will be equal to the parent's ones immediately
after the fork(). But the actual physical memory has only one copy
resident, although ps will show it on both processes (at least I think
that's how ps works). Of course later they will diverge.

>RS> My understanding is that COW is used when forking (on Linux).  

I think this is true of all modern Unix systems.

>RS> So maybe "ps v pid" is reflecting that. If that's the case, is
>RS> there a way to better determine the child's memory usage?
 
Define `memory usage' in the light of the above.

As long as the parent is still around and you don't run out of virtual
memory in the child, not much harm is done.

If the parent stops and you don't run out of virtual memory in the
child, the excessive pages will eventually be paged out, and then no
longer occupy physical memory. As long as you have enough swap space it
shouldn't be a big problem. The extra paging activity is a bit of a
loss, however.

If you run out of virtual memory in the child you have a problem, however.

>RS> If it's not the case and I'm using
>RS> modules I don't need, how can I reduce the memory usage to what the child
>RS> actually uses instead of including everything the parent is using?

The best would be to fork the child before you import the excess modules
in the parent. If that is not possible you could try to delete as much
in the child as you can, for example by
del wx; del sys.modules['wx'] etc, delete all variables that you don't
need, and hope the garbage collector will clean up enough. But it will
make you application quite complicated. From the python level you can't
get rid of loaded shared libraries, however. And trying to do that from
the C level is probably close to committing suicide.

My advise: don't worry until you really experience memory problems.
-- 
Piet van Oostrum <piet at cs.uu.nl>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: piet at vanoostrum.org