[Python-bugs-list] [ python-Bugs-445862 ] bsddb fails for larger amount of data
noreply@sourceforge.net
noreply@sourceforge.net
Fri, 03 Aug 2001 14:40:03 -0700
Bugs item #445862, was opened at 2001-07-30 00:21
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=445862&group_id=5470
Category: Extension Modules
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: bsddb fails for larger amount of data
Initial Comment:
The attached script fails after approx. 72500 insert
operations. If you vary the size of the keys and/or
the values, the bug occurs earlier or later, but even
with a value size of 1 the bug will occur. Probably,
this explains also bug #408271 ("crash in shelve
module").
Platform: W2K
----------------------------------------------------------------------
>Comment By: Tim Peters (tim_one)
Date: 2001-08-03 14:40
Message:
Logged In: YES
user_id=31435
Thanks for taking a look, Skip! On Win98SE it dies for me
like so:
...
70000
71000
72000
Last i: 72758, last key:abcdef1691515.8934
Traceback (most recent call last):
File "ka.py", line 15, in ?
db[key] = val
bsddb.error: (0, 'Error')
test.dbm is 37,778,944 bytes at the end. I assume
Anonymous has the same problem (if not, he/she should say
so).
On Windows we use the ancient db.1.85.win32.zip, from
the "bsd db" (not "bsddb"!) link at
http://www.nightmare.com/software.html
I doubt Sam has done any maintenance on that in years; and
afraid I don't know anything else about this.
----------------------------------------------------------------------
Comment By: Skip Montanaro (montanaro)
Date: 2001-08-03 13:25
Message:
Logged In: YES
user_id=44345
What version of libdb are you using? I'm running your
script on Linux at the moment. I had to change it slightly
because the only machine I have available with the spare
cojones to run that script is running 1.5.2 (so I call
random.uniform instead of using a Random instance). On that
machine I'm sort of ashamed to say I'm still running the
known buggy libdb 1.85. So far I'm up to 680,000 keys with
a db file of over 166MB with no problem. On my laptop
running 2.1 and libdb3 (and a much more modestly performing
disk drive) I gave up after about 287,000 keys.
I then changed the db open call to bsddb.btopen and watched
it march (slowly) up to 183,000 keys and a 32MB file on
disk before I killed it. Aside from the grief it gives my
disk drives, I don't see anything particularly bad
happening.
You didn't include a traceback with your bug report. What
was printed? Perhaps it's something simple like running
out of disk space. In any case, I think trying to create a
libdb database of 1,000,000 sort of random keys is going to
strain that package and most disk drives in any case, bugs
or no bugs.
My guess is that if there's a bug it's in libdb, not the
bsddb module.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-08-03 00:50
Message:
Logged In: NO
Here it is:
import anydbm
import bsddb
import random
MAX = 1000000
r = random.Random(42)
r.seed(1017)
db = anydbm.open("test.dbm", "n")
#db = bsddb.hashopen("test.dbm", "n")
try:
for i in xrange(0, MAX):
if i % 1000 == 0: print i
key = "abcdef" + str(r.uniform(0, 10 * MAX))
val = "a" * 80 + str(i)
db[key] = val
finally:
db.close()
print "Last i: %s, last key:%s" % (i,key)
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-08-02 12:41
Message:
Logged In: YES
user_id=31435
Alas, there's no script attached -- please attach one, so
we have something concrete to investigate.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-08-02 03:08
Message:
Logged In: NO
I was getting crashes in shelve module, Using NT4 (Python
2.1 (#15, Apr 16 2001, 18:25:49) [MSC 32 bit (Intel)] on
win32). I've changed my program to re-read previously
written keys fairly frequently, and I get keyerrors for
keys that have definitely been written, and that gave no
error a little earlier in the same program. The program
doesn't contain any delete statements.
The same program works when using dumbdbm instead of bsddb
(but produces huge indexes), so there definitely appears to
be a problem with bsddbm on windows NT.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=445862&group_id=5470