[Python-Dev] Introducing memprof (was PyErr_NoMemory)
Vladimir Marangozov
Vladimir.Marangozov@inrialpes.fr
Fri, 18 Aug 2000 21:09:48 +0200 (CEST)
[Tim, on PyErr_NoMemory]
>
> Looks good to me. And if it breaks something, it will be darned hard to
> tell <wink>.
It's easily demonstrated with the memprof.c module I'd like to introduce
quickly here.
Note: I'll be out of town next week and if someone wants to
play with this, tell me what to do quickly: upload a (postponed) patch
which goes in pair with obmalloc.c, put it in a web page or remain quiet.
The object allocator is well tested, the memory profiler is not so
thouroughly tested... The interface needs more work IMHO, but it's
already quite useful and fun in it's current state <wink>.
Demo:
~/python/dev>python -p
Python 2.0b1 (#9, Aug 18 2000, 20:11:29) [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
Copyright 1995-2000 Corporation for National Research Initiatives (CNRI)
>>>
>>> # Note the -p option -- it starts any available profilers through
... # a newly introduced Py_ProfileFlag. Otherwise you'll get funny results
... # if you start memprof in the middle of an execution
...
>>> import memprof
>>> memprof.__doc__
'This module provides access to the Python memory profiler.'
>>>
>>> dir(memprof)
['ALIGNMENT', 'ERROR_ABORT', 'ERROR_IGNORE', 'ERROR_RAISE', 'ERROR_REPORT', 'ERROR_STOP', 'MEM_CORE', 'MEM_OBCLASS', 'MEM_OBJECT', '__doc__', '__name__', 'geterrlevel', 'getpbo', 'getprofile', 'getthreshold', 'isprofiling', 'seterrlevel', 'setpbo', 'setproftype', 'setthreshold', 'start', 'stop']
>>>
>>> memprof.isprofiling()
1
>>> # It's running -- cool. We're now ready to get the current memory profile
...
>>> print memprof.getprofile.__doc__
getprofile([type]) -> object
Return a snapshot of the current memory profile of the interpreter.
An optional type argument may be provided to request the profile of
a specific memory layer. It must be one of the following constants:
MEM_CORE - layer 1: Python core memory
MEM_OBJECT - layer 2: Python object memory
MEM_OBCLASS - layer 3: Python object-specific memory
If a type argument is not specified, the default profile is returned.
The default profile type can be set with the setproftype() function.
>>>
>>> mp = memprof.getprofile()
>>> mp
<global memory profile, layer 2, detailed in 33 block size classes>
>>>
>>> # now see how much mem we're using, it's a 3 tuple
... # (requested mem, minimum allocated mem, estimated mem)
...
>>> mp.memory
(135038, 142448, 164792)
>>> mp.peakmemory
(137221, 144640, 167032)
>>>
>>> # indeed, peak values are important. Now let's see what this gives in
... # terms of memory blocks
...
>>> mp.blocks
(2793, 2793)
>>> mp.peakblocks
(2799, 2799)
>>>
>>> # Again this is a 2-tuple (requested blocks, allocated blocks)
... # Now let's see the stats of the calls to the allocator.
...
>>> mp.malloc
(4937, 0, 0)
>>> mp.calloc
(0, 0, 0)
>>> mp.realloc
(43, 0, 0)
>>> mp.free
(2144, 0, 0)
>>>
>>> # A 3-tuple (nb of calls, nb of errors, nb of warnings by memprof)
... #
... # Good. Now let's see the memory profile detailed by size classes
... they're memory profile objects too, similar to the global profile:
>>>
>>> mp.sizeclass[0]
<size class memory profile, layer 2, block size range [1..8]>
>>> mp.sizeclass[1]
<size class memory profile, layer 2, block size range [9..16]>
>>> mp.sizeclass[2]
<size class memory profile, layer 2, block size range [17..24]>
>>> len(mp.sizeclass)
33
>>> mp.sizeclass[-1]
<size class memory profile, layer 2, block size range [257..-1]>
>>>
>>> # The last one is for big blocks: 257 bytes and up.
... # Now let's see ithe detailed memory picture:
>>>
>>> for s in mp.sizeclass:
... print "%.2d - " % s.sizeclass, "%8d %8d %8d" % s.memory
...
00 - 0 0 0
01 - 3696 3776 5664
02 - 116 120 160
03 - 31670 34464 43080
04 - 30015 32480 38976
05 - 10736 11760 13720
06 - 10846 11200 12800
07 - 2664 2816 3168
08 - 1539 1584 1760
09 - 1000 1040 1144
10 - 2048 2112 2304
11 - 1206 1248 1352
12 - 598 624 672
13 - 109 112 120
14 - 575 600 640
15 - 751 768 816
16 - 407 408 432
17 - 144 144 152
18 - 298 304 320
19 - 466 480 504
20 - 656 672 704
21 - 349 352 368
22 - 542 552 576
23 - 188 192 200
24 - 392 400 416
25 - 404 416 432
26 - 640 648 672
27 - 441 448 464
28 - 0 0 0
29 - 236 240 248
30 - 491 496 512
31 - 501 512 528
32 - 31314 31480 31888
>>>
>>> for s in mp.sizeclass:
... print "%.2d - " % s.sizeclass, "%8d %8d" % s.blocks
...
00 - 0 0
01 - 236 236
02 - 5 5
03 - 1077 1077
04 - 812 812
05 - 245 245
06 - 200 200
07 - 44 44
08 - 22 22
09 - 13 13
10 - 24 24
11 - 13 13
12 - 6 6
13 - 1 1
14 - 5 5
15 - 6 6
16 - 3 3
17 - 1 1
18 - 2 2
19 - 3 3
20 - 4 4
21 - 2 2
22 - 3 3
23 - 1 1
24 - 2 2
25 - 2 2
26 - 3 3
27 - 2 2
28 - 0 0
29 - 1 1
30 - 2 2
31 - 2 2
32 - 51 51
>>>
>>> # Note that just started the interpreter and analysed it's initial
... # memory profile. You can repeat this game at any point of time,
... # look at the stats and enjoy a builtin memory profiler.
... #
... # Okay, now to the point on PyErr_NoMemory: but we need to restart
... # Python without "-p"
>>>
~/python/dev>python
Python 2.0b1 (#9, Aug 18 2000, 20:11:29) [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
Copyright 1995-2000 Corporation for National Research Initiatives (CNRI)
>>>
>>> import memprof
>>> memprof.isprofiling()
0
>>> memprof.start()
memprof: freeing unknown block (0x40185e60)
memprof: freeing unknown block (0x40175098)
memprof: freeing unknown block (0x40179288)
>>>
>>> # See? We're freeing unknown blocks for memprof.
... # Okay, enough. See the docs for more:
...
>>> print memprof.seterrlevel.__doc__
seterrlevel(flags) -> None
Set the error level of the profiler. The provided argument instructs the
profiler on how tolerant it should be against any detected simple errors
or memory corruption. The following non-exclusive values are recognized:
ERROR_IGNORE - ignore silently any detected errors
ERROR_REPORT - report all detected errors to stderr
ERROR_STOP - stop the profiler on the first detected error
ERROR_RAISE - raise a MemoryError exception for all detected errors
ERROR_ABORT - report the first error as fatal and abort immediately
The default error level is ERROR_REPORT.
>>>
>>> # So here's you're PyErr_NoMemory effect:
...
>>> memprof.seterrlevel(memprof.ERROR_REPORT | memprof.ERROR_RAISE)
>>>
>>> import test.regrtest
memprof: resizing unknown block (0x82111b0)
memprof: raised MemoryError.
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "./Lib/test/regrtest.py", line 39, in ?
import random
File "./Lib/random.py", line 23, in ?
import whrandom
File "./Lib/whrandom.py", line 40, in ?
class whrandom:
MemoryError: memprof: resizing unknown block (0x82111b0)
>>>
>>> # Okay, gotta run. There are no docs for the moment. Just the source
... and function docs. (and to avoid another exception...)
>>>
>>> memprof.seterrlevel(memprof.ERROR_IGNORE)
>>>
>>> for i in dir(memprof):
... x = memprof.__dict__[i]
... if hasattr(x, "__doc__"):
... print ">>>>>>>>>>>>>>>>>>>>>>>>>>>>> [%s]" % i
... print x.__doc__
... print '='*70
...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [geterrlevel]
geterrlevel() -> errflags
Get the current error level of the profiler.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [getpbo]
getpbo() -> int
Return the fixed per block overhead (pbo) used for estimations.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [getprofile]
getprofile([type]) -> object
Return a snapshot of the current memory profile of the interpreter.
An optional type argument may be provided to request the profile of
a specific memory layer. It must be one of the following constants:
MEM_CORE - layer 1: Python core memory
MEM_OBJECT - layer 2: Python object memory
MEM_OBCLASS - layer 3: Python object-specific memory
If a type argument is not specified, the default profile is returned.
The default profile type can be set with the setproftype() function.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [getthreshold]
getthreshold() -> int
Return the size threshold (in bytes) between small and big blocks.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [isprofiling]
isprofiling() -> 1 if profiling is currently in progress, 0 otherwise.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [seterrlevel]
seterrlevel(flags) -> None
Set the error level of the profiler. The provided argument instructs the
profiler on how tolerant it should be against any detected simple errors
or memory corruption. The following non-exclusive values are recognized:
ERROR_IGNORE - ignore silently any detected errors
ERROR_REPORT - report all detected errors to stderr
ERROR_STOP - stop the profiler on the first detected error
ERROR_RAISE - raise a MemoryError exception for all detected errors
ERROR_ABORT - report the first error as fatal and abort immediately
The default error level is ERROR_REPORT.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [setpbo]
setpbo(int) -> None
Set the fixed per block overhead (pbo) used for estimations.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [setproftype]
setproftype(type) -> None
Set the default profile type returned by getprofile() without arguments.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [setthreshold]
setthreshold(int) -> None
Set the size threshold (in bytes) between small and big blocks.
The maximum is 256. The argument is rounded up to the ALIGNMENT.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [start]
start() -> None
Start the profiler. If it has been started, this function has no effect.
======================================================================
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [stop]
stop() -> None
Stop the profiler. If it has been stopped, this function has no effect.
======================================================================
--
Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252