Populating a dictionary, fast

[Alberto Berti]

>>>>> "Steven" == Steven D'Aprano <steve at REMOVE-THIS-cybersource.com.au> writes:

    Steven> $ time ./slurp_dict.py Starting at Sun Nov 11 14:26:51
    Steven> 2007 Line 0 Line 1000000 Line 2000000 Line 3000000 Line
    Steven> 4000000 Line 5000000 Line 6000000 Line 7000000 Line
    Steven> 8000000 Items in dict: 8191180 Completed import at Sun Nov
    Steven> 11 14:29:31 2007 Starting to delete dict...


    Steven> Traceback (most recent call last): File "./slurp_dict.py",
    Steven> line 20, in <module> del id2name KeyboardInterrupt

    Steven> real 35m52.334s user 1m17.663s sys 0m16.758s


    Steven> Notice that the dict is completely read into memory in
    Steven> just two and a half minutes. The script then tries to
    Steven> delete the dict, and 32 minutes later is still
    Steven> struggling. That's the point I got sick of waiting and
    Steven> interrupted the script.

    Steven> Conclusion: it's a memory issue, or maybe a garbage
    Steven> collection issue, not a problem with dicts.


uh, strange results...

I run your same scripts with and without garbage collection enabled
and those are the results:

with gc enabled:

azazel at lizard:~/wip/zodb_test$ python slurp_dict.py 
Starting at Sun Nov 11 16:35:12 2007
Line 0
Line 1000000
Line 2000000
Line 3000000
Line 4000000
Line 5000000
Line 6000000
Line 7000000
Line 8000000
Items in dict: 8191180
Completed import at Sun Nov 11 16:36:03 2007
Starting to delete dict...
Completed deletion at Sun Nov 11 16:36:09 2007
Finishing at Sun Nov 11 16:36:09 2007

and without gc enabled

azazel at lizard:~/wip/zodb_test$ python slurp_dict.py 
Starting at Sun Nov 11 16:39:02 2007
Line 0
Line 1000000
Line 2000000
Line 3000000
Line 4000000
Line 5000000
Line 6000000
Line 7000000
Line 8000000
Items in dict: 8191180
Completed import at Sun Nov 11 16:39:49 2007
Starting to delete dict...
Completed deletion at Sun Nov 11 16:39:56 2007
Finishing at Sun Nov 11 16:39:56 2007


all with python2.4 on and i386 Linux

cheers 

Alberto