[Python-bugs-list] [ python-Bugs-229810 ] Memore leak in pickle and cPickle
noreply@sourceforge.net
noreply@sourceforge.net
Tue, 24 Apr 2001 16:47:01 -0700
Bugs item #229810, was updated on 2001-01-23 07:28
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=229810&group_id=5470
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Vladimir Kralik (vlk)
Assigned to: Barry Warsaw (bwarsaw)
Summary: Memore leak in pickle and cPickle
Initial Comment:
# When Pickler.object is used for dump typles into file and Unpickler for
# load from files. A loaded object are not garbage collected.
# When function dump(object,file) is used Unpickler works fine.
# Problem is in pickle and cPickle module
# tested on Python 2.0 Red Hat Linux 6.2
import cPickle
#import pickle
import gc
f=open("xxx","w")
pic=cPickle.Pickler(f,1) # ERROR
#pic=pickle.Pickler(f,1) # ERROR
for i in range(100):
#cPickle.dump(([long(i),long(i),long(i),long(i)],i),f)
# this is OK
#pickle.dump(([long(i),long(i),long(i),long(i)],i),f)
# this is OK
pic.dump(([long(i),long(i),long(i),long(i)],i))
# Memory leak
f.close()
gc.set_debug(gc.DEBUG_STATS)
f=open("xxx","r")
u=cPickle.Unpickler(f)
try:
while 1:
gc.collect()
print u.load()
except EOFError:
pass
f.close()
----------------------------------------------------------------------
Comment By: Naris Siamwalla (naris)
Date: 2001-04-24 16:47
Message:
Logged In: YES
user_id=67995
barry's python snippet memory leaks on python 2.1 final
on rh6.2.
----------------------------------------------------------------------
Comment By: Barry Warsaw (bwarsaw)
Date: 2001-02-23 10:59
Message:
# When Pickler.object is used for dump typles into file and Unpickler for load
# from files. A loaded object are not garbage collected. When function
# dump(object,file) is used Unpickler works fine. Problem is in pickle and
# cPickle module tested on Python 2.0 Red Hat Linux 6.2
import gc
def main():
fp = open('/tmp/xxx', 'w')
pic = pickle.Pickler(fp, 1) # ERROR
for i in range(10000):
pickle.dump(([long(i), long(i), long(i), long(i)], i), fp)
# this is OK
pic.dump(([long(i), long(i), long(i), long(i)], i))
# Memory leak
fp.close()
gc.set_debug(gc.DEBUG_STATS)
fp = open('/tmp/xxx')
upic = pickle.Unpickler(fp)
try:
while 1:
gc.collect()
print upic.load()
except EOFError:
pass
fp.close()
if __name__ == '__main__':
import sys
if len(sys.argv) > 1:
import cPickle
pickle = cPickle
else:
import pickle
main()
----------------------------------------------------------------------
Comment By: Barry Warsaw (bwarsaw)
Date: 2001-02-23 10:58
Message:
When cranking up the number of objects placed in the pickle to 10000, I do see some memory growth when unpickling as clocked by top. The cPickle growth is much smaller than the pickle growth, which already appears fairly minimal.
I will investigate further with Insure++.
I don't see how the problem could be related to __del__ since the only thing we're dumping and loading are builtin objects (tuples, lists, longs, and ints).
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-01-23 19:28
Message:
Barry, can you look into this? I would first see if this is really reproducable without using Insure++; somehow it looks a bit fishy. Could also be fixed in 2.1 because now modules participate in gc. Or could have to do with a __del__? Also, I doubt the claim that this is a leak with both pickle and cPickle.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=229810&group_id=5470