bsddb.error

[Romuald Texier]

On linux I also have problems with bsddb that does not occur when I force 
shelve to use gdb instead ; I didn't get further because I did not 
understand why and how. Maybe your problems may disappear if you force gdb.

blobbybirdman wrote:

> Ok, no reply....guess I didn't capture anyones interest, doh! Can anyone
> at least point me in the right direction for more information on the
> shelve module and what the hell might be causing a "bsddb.error: (0,
> 'Error')".
> 
> Dont make me beg, I am pitiful enough as is ;)
> 
> Blobby
> 
> Mark blobby Robinson wrote:
> 
> > Hi,
> >
> > I wonder if anyone can provide some insight to an error I am getting. I
> > am
> > unable to figure out what is causing it. I am a developing some software
> >
> > for a bioinformatics research lab, specifically for the analysis of DNA
> > sequences.
> >
> > Firstly, I am using python 1.5.2 on a machine running red hat linux 7.0.
> >
> > The machine is running at 700Mhz with 380Mb RAM and about 1GB free space
> >
> > on root.
> >
> > In the broadest terms the program is generating a list of "motif"
> > objects and when the length of this list exceeds one million I am
> > shelving the objects. The error that I am getting is that while I am
> > shelving the objects I sometimes receive the following exception
> > bsddb.error:
> >
> > 90% Written
> > 100% Written
> > Traceback (innermost last):
> >   File "runMotif.py", line 34, in ?
> >     filename = genMotifs(posList, S, filename)
> >   File "motif.py", line 161, in genMotifs
> >     motifs = writeMotifs(motiflist, filename, S)
> >   File "motif.py", line 189, in writeMotifs
> >     motifDict[hstring] = motif(x, S)
> >   File "/usr/lib/python1.5/shelve.py", line 71, in __setitem__
> >     self.dict[key] = f.getvalue()
> > bsddb.error: (0, 'Error')
> >
> > Ok, now I'll be a bit more specific about what I am doing at the
> > shelving step! Each of the  "motif" objects in the list contains 3
> > attributes which contain strings of length 5 to 10. I use these strings
> > to generate a unique key for each motif in the shelved data file. Then
> > for each object in the list I perform a check to see if that key has
> > been generated previously. If key is not found in shelved file then
> > I add the object, else I simply update the shelved object accordingly.
> > The code is as follows:
> >
> > def writeMotifs(motiflist, filename, S):
> >      motifDict = shelve.open(filename)
> >      print "writing to %s" % filename
> >      count = 0
> >      for x in motiflist:
> >           hstring = x[0][0] + '_' + x[1][0] + '_' + x[2][0]
> >           if count % 100000 == 0:
> >                print "%s%% Written" % (count/10000)
> >           if motifDict.has_key(hstring):
> >                temp = motifDict[hstring]
> >                if checkNoMotifOverlap(temp, x) is not 1:
> >                     temp.setCount()
> >                     temp.setPos(x)
> >                     motifDict[hstring] = temp
> >               else:
> >                    motifDict[hstring] = motif(x, S)
> >               count = count + 1
> >      length = len(motifDict)
> >      motifDict.close
> >  return length
> >
> > This program is taking a huge combinatorial problem and is therefore
> > producing absolutely piles of data. I have checked to ensure that I am
> > not running out of RAM during execution, and although the shelved data
> > file is getting pretty big I have so far only used run the analysis on
> > short sequences producing output in the region of 90-160MB size. The
> > fact that it falls over at different sizes suggests to me that I am not
> > hitting some linux defined constant for file sizes or the like, but
> > I can't think what it could be. I have check the number of objects in
> > the shelved file and it has ranged between 90,000 and 143, 000 before it
> >
> > keels over and dies.
> >
> > Can anyone suggest any reason this problem might be occuring?
> >
> > cheers
> >
> > Blobby
> 

-- 
Romuald Texier