Need design advice. What's my best approach for storing this data?

[Rene Pijlman]

Mudcat:
>My initial thought was to put the data in large dictionaries and shelve
>them (and possibly zipping them to save storage space until the data is
>needed). However, these are huge files.

ZODB solves that problem for you.
http://www.zope.org/Wikis/ZODB/FrontPage

More in particular "5.3 BTrees Package":
http://www.zope.org/Wikis/ZODB/guide/node6.html#SECTION000630000000000000000

But I've only used ZODB for small databases compared to yours. It's
supposed to scale very well, but I can't speak from experience.

-- 
René Pijlman