Shelve and Object Size

[Putland, Karl]

> -----Original Message-----
> From: Curtis Jensen [mailto:cjensen at bioeng.ucsd.edu]
> Sent: Thursday, August 10, 2000 2:28 PM
> To: python-list at python.org
> Subject: Shelve and Object Size
> 
> 
> Is there a limit on the size of an object that shelve can handle?  For
> example:
> 
> >>> import shelve
> >>> from Numeric import *
> >>> writer = shelve.open ( 'my_db' )
> >>> writer['2'] = zeros([2])
> >>> writer['big'] = zeros([1000000])
> Traceback (innermost last):
>   File "<stdin>", line 1, in ?
>   File "/usr/local/Python/lib/python1.5/shelve.py", line 71, in
> __setitem__
>     self.dict[key] = f.getvalue()
> dbm.error: Cannot add item to database
> 
> I need to but some rather large Python objects into a database.  I
> thought shelve would be the easiest way.  How should I do this?  Can I
> make shelve work, or should I use something different?  Thanks.
> 
> -- 
> Curtis Jensen
> 

I also have questions about shelves.

I'm currently using Python's bsddb through shelves to cache data from Oracle
and M$SQL in transit to a ReporLab generated PDF on WinNT4.  I've been
having issues when the shelves approach or exceed 1GB.  A couple of
questions...

[begin]
import shelve
s = shelve.open('test','n')
for i in xrange(100000000):
    error = 0
    while error < 10:
        try:
            s[str(i)] = 'I want to be enourmous'
            error = 10
        except:
            print 'coughed on',i
            error = error + 1
            if error == 10:
                print "couldn't insert",i
[end]


The above script on my machine inserted 474003 items into a file that wound
up at 1,084,448,768 bytes.  Are there any know issues with bsddb and shelve
that limit file size or number of key:value pairs?

--Karl Putland