huge persistent dictionary

[Quinn Dunkan]

On Tue, 07 Nov 2000 13:05:10 -0000, gunter  <gb at a3design.de> wrote:
>Hello!
>
>My Problem is, that I want to store an huge amount of data in an
>dictionary and to make it persistent.
...
>PROBLEM:
>compiling this large file "data.py" takes about 5 Minutes and takes
>about 50 Megs of Memory.
>
>QUESTION:
>Is there any other way to make such structures of primitive types
>persistent?
>The compiling-trick-thing is a little rude, i think!

Others have made some good suggestions, but I'd recommend you try marshal
first.  It only works on primitive types, but it's faster than cPickle.  And
it's essentially the same as your import trick (it writes and reads python
bytecode), delta the huge intermediate file.

still-waiting-for-someone-to-post-a-mmap-solution-ly y'rs