memory efficient set/dictionary

[koara]

Hello Steven,

On Jun 10, 5:29 pm, Steven D'Aprano
<s... at REMOVE.THIS.cybersource.com.au> wrote:
> > ...
> How do you know it won't fit in main memory if you don't know the
> overhead? A guess? You've tried it and your computer crashed?

exactly

> > Please recommend a module that allows persistent set/dict storage +
> > fast query that best fits my problem,
>
> Usually I love guessing what people's problems are before making a
> recommendation, but I'm feeling whimsical so I think I'll ask first.
>
> What is the problem you are trying to solve? How many keys do you have?

Corpus processing. There are in the order of billions to tens of
billions keys (64bit integers).

> Can you group them in some way, e.g. alphabetically? Do you need to search
> on random keys, or can you queue them and do them in the order of your
> choice?


Yes, keys in sets and dictionaries can be grouped in any way, order
doesn't matter. Not sure what you mean.
Yes, i need fast random access (at least i do without having to
rethink and rewrite everything, which is what i'd like to avoid with
the help of this thread :-)

Thanks for the reply!