sparse sets of integers?

[François Pinard]

[runes]

> You have sets in Python 2.3x and 2.4x. I don't know if they can handle
> your amounts of data, but i guess so.

The original poster (OP) spoke about 10**13 numbers, if I remember
correctly (I did not keep the original message).  It all depends on the
real sparsity of the spare sets! :-) If not sparse enough, this might be
a bit inordinate for standard Python sets on most machines, also given
that each Python integer uses a lot more than 4 bytes...

On the other hand, the OP also wrote that the numbers he handles are
grouped in big and compact intervals, separated by big unpopulated
holes.  So maybe his data could be represented in Python as a
user-written type hiding a sorted list of intervals, for which it should
be relatively easy to write set functions.  That might make the problem
tractable enough, with reasonably speedy processing.

As last resort, the wanted sets could be represented as sorted files of
numbers.  Slower, but set operations would also be easy to write.

-- 
François Pinard   http://pinard.progiciels-bpi.ca