very large dictionary

[Enrico Franchi]

Simon Strobl <Simon.Strobl at gmail.com> wrote:

> Well, as I was using Python, I did not expect to have to care about
> the language's internal affairs that much. I thought I could simply do
> always the same no matter how large my files get. In other words, I
> thought Python was really scalable.

It's not Python here. It's just how computers work. IMHO having a
gargantuan dictionary in memory is not a good idea (unless explicitly
proven otherwise): this is the kind of job databases have been created
for. 

Besides, this is not a matter of Python. If you were using C or another
language, I would have sugested to use databases in order to manipulate
GB's of data. Luckily enought, using databases in Python is far easier
than doing so in C/C++/Java. *And* there are thin abstractions over
databases so you don't even need to know how to use them (though, I
suggest that you *do* learn something about DB's and expecially
relational DB's, SQL is not *that* bad). 


-- 
-riko