very large dictionary

[Steven D'Aprano]

On Tue, 05 Aug 2008 01:20:08 -0700, Simon Strobl wrote:


>> > I thought it would be practical not to create the dictionary from a
>> > text file each time I needed it. I.e. I thought loading the .pyc-file
>> > should be faster. Yet, Python failed to create a .pyc-file
>>
>> Probably a good example of premature optimization.
> 
> Well, as I was using Python, I did not expect to have to care about the
> language's internal affairs that much. I thought I could simply do
> always the same no matter how large my files get. In other words, I
> thought Python was really scalable.

Yeah, it really is a pain when abstractions leak.

http://www.joelonsoftware.com/articles/LeakyAbstractions.html


>> Out of curiosity, how
>> long does it take to create it from a text file?
> 
> I do not remember this exactly. But I think it was not much more than an
> hour.

Hmmm... longer than I expected. Perhaps not as premature as I thought. 
Have you tried the performance of the pickle and marshal modules?



-- 
Steven