speeding up dictionary creation

[Peter Schneider-Kamp]

[Bob van der Poel in his original post]
> 
> it appears to me that when reading/parsing the initial datafile the
> majority of the time is taken in creating new entries, not in parsing
> the file. Is there a way to preallocate entries or otherwise speed this
> up. On a short set to data there doesn't seem to be a problem, but as
> the list increases in size the load time goes up in what appears to be
> N^2.

I do not think the majority is spent in creating new entries. See the
statistics below - creating the dict from the list is fast.
(Especially look at the second statistic ...)

[first statistic on 17576 dict entries - worst of three runs]
Creating list of 17576 keys ...
         37262 function calls (30 primitive calls) in 2.680 CPU seconds

Creating dictionary from list of keys ...
         2 function calls in 0.080 CPU seconds

pickle.dump of dictionary ...
         123041 function calls (87889 primitive calls) in 11.960 CPU
seconds

binary pickle.dump of dictionary ...
         123041 function calls (87889 primitive calls) in 12.620 CPU
seconds

cPickle.dump of dictionary ...
         2 function calls in 0.800 CPU seconds

binary cPickle.dump of dictionary ...
         2 function calls in 0.400 CPU seconds

pickle.load of dictionary ...
         105466 function calls in 12.630 CPU seconds

binary pickle.load of dictionary ...
         52739 function calls in 7.320 CPU seconds

cPickle.load of dictionary ...
         17578 function calls in 4.650 CPU seconds

binary cPickle.load of dictionary ...
         2 function calls in 0.540 CPU seconds

******************************************************

[second statistic on cPickle with 456976 dict entries]

Creating list of 456976 keys ...
         968790 function calls (30 primitive calls) in 69.300 CPU
seconds

Creating dictionary from list of keys ...
         2 function calls in 2.200 CPU seconds

binary cPickle.dump of dictionary ...
         2 function calls in 1.890 CPU seconds

binary cPickle.load of dictionary ...
         2 function calls in 9.620 CPU seconds

hope-that-helps-your-decision-ly y'rs
Peter
-- 
Peter Schneider-Kamp          ++47-7388-7331
Herman Krags veg 51-11        mailto:peter at schneider-kamp.de
N-7050 Trondheim              http://schneider-kamp.de