I'm surprised you didn't recommend to use ZODB. Seems like an ideal way to manage this large amount of data as a collection of Python objects...