Shelve problem: why so big?

[Ryan Wendt]

I have an idea, but considering the fact that I just learned python
yesterday, it probably won't help. Is it possible that it could be storing
each number as larger ubit, such as a word, or even dword? That could be a
possible explanation, though I don't know much about defining words or
dwords in python.

"Geert-Jan Giezeman" <geert at cs.uu.nl> wrote in message
news:9sdol4$2te$1 at samos.cs.uu.nl...
>
> I am trying to create a shelve from a table of 1100 by 1100 floating
> point numbers, which I want to store as 1100 lists of length 1100,
> indexed by a key. Unfortunately, I get an error when I try to create
> this:
> bsddb.error; (22, Invalid argument)
>
> When I store smaller lists (I tried up to size 1100*600), there is no
> problem.
>
> My guess is that this has to do with the file size. Even with the
> shorter list the size of the file is 1.01 GB (1,090,331,648 bytes).
> Continuing in guessing mode, it could be that at a certain moment the
> size of the file is doubled and that this fails. (I don't know exactly
> how large files can get on an NTFS file system, but a 2GB maximum does
> not sound unreasonable).
>
> Now on to the questions:
>
> 1) Are those large filesizes for shelves normal? A bare minimum for
> storing the data would be 8*1100*1100 = 9,681,000 bytes. So pickling
> and database overhead would be a factor 100.
>
> 2)Are there ways to circumvent those problems? Use a different
> database (I like shelves, though), ...