[Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

[Jesus Cea]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/11 06:46, Nick Coghlan wrote:
> The server actually looks it may be in a generally unhappy state, 
> perhaps due to previous builds that failed without cleaning up
> after themselves properly. How many python processes do you see
> hanging around?

Just now:

"""
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU
PROCESS/NLWP
  4340 buildbot  366M  344M sleep    1    0   0:11:23 0.0% python/2
 10097 buildbot  366M 8096K sleep   25    0   0:00:00 0.0% python/1
 10099 buildbot  366M 8100K sleep   25    0   0:00:00 0.0% python/1
 10098 buildbot  366M 8108K sleep   26    0   0:00:00 0.0% python/1
 27698 buildbot  251M 5244K sleep    1    0   0:00:00 0.0% python/1
 27697 buildbot  251M   11M sleep    1    0   0:00:00 0.0% python/1
 27695 buildbot  251M 5852K sleep    1    0   0:00:00 0.0% python/1
 27694 buildbot  251M 5844K sleep    1    0   0:00:00 0.0% python/1
 27696 buildbot  251M 5884K sleep    1    0   0:00:00 0.0% python/1
 27693 buildbot  251M 5964K sleep    1    0   0:00:00 0.0% python/1
  9893 buildbot  202M  198M sleep    1    1   0:09:32 0.0% python/2
 14538 buildbot  200M 4700K sleep    1    1   0:00:00 0.0% python/1
 25971 buildbot  194M  189M sleep   10    0   0:11:22 0.0% python/2
  2616 buildbot  120M  114M sleep    1    0   0:06:38 0.0% python/47
 11204 buildbot  118M 5612K sleep    1    0   0:00:00 0.0% python/2
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE

    23       56 4073M 1632M    40%   0:39:38 0.0%
pythonbuildbot.uk.openindiana.org
"""

This particular build seems to have hang, usual result of running out
of memory. Note the SWAP usage of 4073MB, when my limit seems to be
4096MB.

The buildbot master process will kill this "hang" processes after the
usual timeout.

I have requested raising my memory limit to my host, with no effect so
far. Anyway, eating >4GB of RAM seems quite overkill.

Doing a "make test" manually I can see the python process doing the
test to eat more than 200MB of RAM, but it only launch a python
process, not a handful like the regular buildbot.

I have verified that the memory use is atribuible to the buildbot,
since if I kill the buildbot processes, my RAM+SWAP usage is negligible.

Thanking for helping me with this.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTmWorplgi5GaxT1NAQIb7QP/Y/Mr0RhhRTM1Rld7xKqNi77tcB0+p4CX
EZ0fViNr/NF6NibKMzowi0pr42iZ3dXN4/yRQgNsvGhfzTrpi+J3Z1GCg5vnqox3
jOC+DQ5IrZylLV+zH46K9j2UJ+4hvU3PWBZcGAt6iB4EVK1h8mvBBW08VeDoN5Cj
Nkqth694BcY=
=KAwa
-----END PGP SIGNATURE-----