Populating a dictionary, fast [SOLVED SOLVED]

Michael Bacarella mbac at gpshopper.com
Thu Nov 15 15:51:25 EST 2007


> On Nov 15, 2:11 pm, Istvan Albert <istvan.alb... at gmail.com> wrote:
> > There is nothing wrong with neither creating nor deleting
> > dictionaries.
> 
> I suspect what happened is this: on 64 bit
> machines the data structures for creating dictionaries
> are larger (because pointers take twice as much space),
> so you run into memory contention issues sooner than
> on 32 bit machines, for similar memory sizes.
> If there is something deeper going
> on please correct me, I would very much like to know.

Since some people missed the EUREKA!, here's the executive summary:
 
	Python2.3: about 45 minutes
	Python2.4: about 45 minutes
	Python2.5: about _30 seconds_

The cut/paste of the EUREKA MOMENT from earlier in this thread:
 
	> > You can download the list of keys from here, it's 43M gzipped:
	> >     http://www.sendspace.com/file/9530i7
	> >
	> > and see it take about 45 minutes with this:
	> >
	> > $ cat cache-keys.py
	> > #!/usr/bin/python
	> > v = {}
	> > for line in open('keys.txt'):
	> >         v[long(line.strip())] = True
	> >
	> >
	> It takes about 20 seconds for me. It's possible it's related to
	> int/long
	> unification - try using Python 2.5. If you can't switch to 2.5,
try
	> using string keys instead of longs.

	Yes, this was it.  It ran *very* fast on Python v2.5.

	Terribly on v2.4, v2.3.

	(I thought I had already evaluated v2.5 but I see now that the
server
	With 2.5 on it invokes 2.3 for 'python'.)

	Thanks!




More information about the Python-list mailing list