Shelve problem: why so big?

[Charles Allen]

> "Geert-Jan Giezeman" <geert at cs.uu.nl> wrote in message
> news:9sdol4$2te$1 at samos.cs.uu.nl...
>>
>> I am trying to create a shelve from a table of 1100 by 1100 floating
>> point numbers, which I want to store as 1100 lists of length 1100,
>> indexed by a key. Unfortunately, I get an error when I try to create
>>...
>> 2)Are there ways to circumvent those problems? Use a different
>> database (I like shelves, though), ...

Let's take a step back: Is the "1100 lists of length 1100" the way you
*want* to access the data internally?  Have you looked at Numeric
arrays?  Those are much cleaner for accessing the data in array format
(slicing in any dimension, faster operations, etc).

If Numeric arrays suit your purposes, you can try using their
toString() method to convert them to a monolithic entity before doing
the shelve.  You would lose the dimension info unless you stuck the
data string inside, e.g., a class that also contained the dimension
info.  I use this to access externally supplied floating point array
data.

You should also look at the NetCDF file format interface in the
Scientific Python module.  This has the advantage of making the data
accessible from many different languages and platforms.  I've never
used it for keyed access on large data sets like you mention, so I
don't know what the lookup speed is like.  Test it and let us know!

-- 
Charles Allen <charles.allen at ieee.org>