Python 2.0b1 List comprehensions are slow

[Kevin O'Connor]

Hello,

I've been using python and advocating it for several years now.  However, I
haven't been keeping up with all the new features proposed on c.l.py.  (Or
the legal battles either?)  After reading about the new features in the 2.0
beta, I knew I had to download a copy and take a look.

First of all, the list comprehension feature has to be one of the NICEST
features to be added to the language.  After reading about it, and writing
some test programs I was one step short of killing off all those ugly map
and operator calls in all my old programs.  The comprehensions are much
easier to read than a mangled map setup, and they have the potential to be
just as fast and significantly more flexible.

Unfortunately, I'm a little concerned about the speed of the new list
feature.  According to the web site, "[a comprehension] is more efficient
than map() with a lambda."  However, this is not true in my test programs -
they show comprehensions to be 50% slower than map with a lambda.

I hope this can be resolved before 2.0 is released.  :-)
-Kevin


Sample output from my program (on a dual Celeron 366, linux 2.4.0-test7) :


Many repeated calls on small lists:
map + operator: 12.11 cpu seconds
map + lambda: 12.82 cpu seconds
comprehensions: 18.59 cpu seconds

One call on a large list:
map + operator: 6.87 cpu seconds
map + lambda: 7.04 cpu seconds
comprehensions: 11.84 cpu seconds


Test program:

#!/usr/bin/env python

import operator
import time

vals = [1, 8, 12, 89, 325, 213, 23, 434, 435, 2435, 5, 45, 423, 435, 324, 5234]

print "Accuracy test:"
print "foo1: %s" % map(operator.add, (12,)*len(vals), vals)
print "foo2: %s" % map(lambda x: x+12, vals)
print "foo3: %s" % [ x+12 for x in vals]
print

def timeit(func):
    st = time.clock()
    func(vals)
    en = time.clock()
    return en - st

counter = range(200000)

def foo1(arry):
    dummyarry = (12,) * len(arry)
    operator__add = operator.add
    __map = map
    for i in counter:
	__map(operator__add, dummyarry, arry)

def foo2(arry):
    __map = map
    func = (lambda x: x+12)
    for i in counter:
	__map(func, arry)

def foo3(arry):
    for i in counter:
	[ x+12 for x in arry]

print "Many repeated calls on small lists:"
print "map + operator: %s cpu seconds" % timeit(foo1)
print "map + lambda: %s cpu seconds" % timeit(foo2)
print "comprehensions: %s cpu seconds" % timeit(foo3)
print

def bar1(arry):
    return map(operator.add, (12,)*len(arry), arry)

def bar2(arry):
    return map(lambda x: x+12, arry)

def bar3(arry):
    return [ x+12 for x in arry]

vals = range(2000000)

print "One call on a large list:"
print "map + operator: %s cpu seconds" % timeit(bar1)
print "map + lambda: %s cpu seconds" % timeit(bar2)
print "comprehensions: %s cpu seconds" % timeit(bar3)
print

-- 
 ------------------------------------------------------------------------
 | Kevin O'Connor                     "BTW, IMHO we need a FAQ for      |
 | koconnor at cse.buffalo.edu            'IMHO', 'FAQ', 'BTW', etc. !"    |
 ------------------------------------------------------------------------