[Python-bugs-list] [ python-Bugs-485139 ] mem leak in interpreter from syntax err
noreply@sourceforge.net
noreply@sourceforge.net
Sat, 08 Dec 2001 15:44:09 -0800
Bugs item #485139, was opened at 2001-11-24 10:45
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=485139&group_id=5470
Category: Python Interpreter Core
Group: Python 2.2
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Neal Norwitz (nnorwitz)
Assigned to: Tim Peters (tim_one)
Summary: mem leak in interpreter from syntax err
Initial Comment:
when a syntax error occurs in the interpreter, the
interpreter leaks
see the attached file for more info
----------------------------------------------------------------------
>Comment By: Tim Peters (tim_one)
Date: 2001-12-08 15:44
Message:
Logged In: YES
user_id=31435
The leaking refs to the integer 1 got plugged in
Python/symtable.c; new revision: 2.9
Closing this now, as there doesn't appear to be any mystery
remaining worth tracking down, and the other mysteries are
too hard to fix.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-08 15:21
Message:
Logged In: YES
user_id=31435
Don't ask how I found this <wink>: in the def and class
examples, we're "leaking" references to the integer 1(!).
C:\Code\python\PCbuild>python_d
Adding parser accelerators ...
Done.
Python 2.2b2+ (#26, Dec 8 2001, 17:33:44) [MSC 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more
information.
>>> def g(): # return the current refcount on 1
... import sys
... return sys.getrefcount(1)
...
[7037 refs]
>>> g() # it's 38 now
38
[7040 refs]
>>> g() # and it's still 38
38
[7040 refs]
>>> def f(): pass # do 3 function defs, and ...
...
[7057 refs]
>>> def f(): pass
...
[7058 refs]
>>> def f(): pass
...
[7059 refs]
>>> g() # ... the refcount on 1 goes up by 3
41
[7060 refs]
>>> class C: pass # then do 5 class defs, and ...
...
[7070 refs]
>>> class C: pass
...
[7071 refs]
>>> class C: pass
...
[7072 refs]
>>> class C: pass
...
[7073 refs]
>>> class C: pass
...
[7074 refs]
>>> g() # ... the refcount on 1 goes up by 5
46
[7075 refs]
>>>
Similarly for, e.g., entering "lambda:42" over and over. I
don't know *why* we're leaking refs to 1 -- it's bizarre.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-08 14:32
Message:
Logged In: YES
user_id=31435
The difference between _Py_RefTotal and the sum of the
refcounts of the objects in refchain is at least partly
accounted for by that static type objects don't appear in
refchain. The two get out of synch already by the time
_Py_RefTotal reaches 4, due to the static PyBaseObject_Type
showing up in a tuple of base classes (thus adding a count
to _Py_RefTotal that's not seen when traversing refchain).
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-07 21:51
Message:
Logged In: YES
user_id=31435
Well, this just gets more mysterious the longer I stare at
it. I added a routine to compute the sum of all the
refcounts in the objects in the refchain, and fiddled
PyRun_InteractiveLoopFlags to print that sum after printing
_Py_RefTotal. That's the first mystery:
C:\Code\python\PCbuild>python_d
Adding parser accelerators ...
Done.
Python 2.2b2+ (#26, Dec 7 2001, 19:25:48) [MSC 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more
information.
>>>
[7004 refs] # _Py_RefTotal
[6540 sum] # sum of ob_refcnt across refchain objs
That is, right off the bat, _Py_RefTotal is 536 larger than
the sum of all refcounts in all known objects. I hoped
they'd be equal.
At least these stay in synch across lots of input
expressions:
>>> 1
1
[7006 refs]
[6542 sum]
Both went up by 2 there.
>>> []
[]
[7011 refs]
[6547 sum]
And both up by 5.
>>> ()
()
[7011 refs]
[6547 sum]
No change.
>>> {}
{}
[7011 refs]
[6547 sum]
No change.
>>> 2.3
2.2999999999999998
[7011 refs]
[6547 sum]
No change. Now comes another Mystery:
>>> def f(): pass
...
[7028 refs]
[6562 sum]
That is, _Py_RefTotal went up by 17, but the sum of all
refcounts only went up by 15. What's up with that?
Then both "leak" 1 for each repetition of a function defn:
>>> def f(): pass
...
[7029 refs]
[6563 sum]
>>> def f(): pass
...
[7030 refs]
[6564 sum]
>>> def f(): pass
...
[7031 refs]
[6565 sum]
>>>
That makes three mysteries. There's no other evidence of
an actual leak, though! In particular,
>>> import sys
[7036 refs]
[6570 sum]
>>> for i in range(10):
... def f(): pass
... print sys.gettotalrefcount()
...
7075
7075
7075
7075
7075
7075
7075
7075
7075
7075
[7037 refs]
[6571 sum]
>>>
That is, _PyRef_Total is *not* going up by 1 on each "def f
(): pass" if the def stmt is inside a loop.
Reducing the priority since this is such a pit.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-07 09:13
Message:
Logged In: YES
user_id=31435
Reopened and assigned to me. I don't care about a leak on
an uncaught syntax error, but something Neal said reminded
me of a problem that's repeatedly wasted my time: "enter
something at a debug-mode interactive prompt repeatedly"
often exhibits "and lose a refcount each time" behavior.
Like so:
>>> 1
1
[7026 refs]
>>> class C: pass
...
[7036 refs]
>>> class C: pass
...
[7037 refs]
>>> class C: pass
...
[7038 refs]
>>> class C: pass
...
[7039 refs]
>>>
Several times this has fooled me into looking for leaks
that don't exist.
>>> def f(): pass
...
[7056 refs]
>>> def f(): pass
...
[7057 refs]
>>> def f(): pass
...
[7058 refs]
>>>
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-07 08:46
Message:
Logged In: YES
user_id=6380
Neal writes:
---
I tried:
for i in range(1000):
try:
import t2
except SyntaxError:
print
But that doesn't leak. I think the reason it doesn't leak
is because
of the try/except. I opened the interpreter and
interactively
created syntax errors (=<return>, =<return>, ...) and it
leaked
for each error, not just once 874 bytes, instead of 38 for
1.
If there is a way to raise a syntax error without exitting
the interpreterr
I think it could blow up faster.
-----
My comment: if it only leaks when you don't catch it, that
doesn't bother me much, since that's the end of the process.
:-) So I'm closing this.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-06 19:11
Message:
Logged In: YES
user_id=6380
Neil, can you provide more evidence? I can't see this leak
in a loop, like this:
| while 1:
| try: compile("/", "", "exec")
| except SyntaxError: pass
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-11-27 12:05
Message:
Logged In: YES
user_id=31435
Assigned to Barry.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=485139&group_id=5470