[Python-ideas] while conditional in list comprehension ??

[Steven D'Aprano]

On 30/01/13 02:44, Wolfgang Maier wrote:

> list(i for i in range(100) if i<50 or stop())
> Really (!) nice (and 2x as fast as using itertools.takewhile())!

I think you are mistaken about the speed. The itertools iterators are highly
optimized and do all their work in fast C code. If you are seeing takewhile
as slow, you are probably doing something wrong: untrustworthy timing code,
misinterpreting what you are seeing, or some other error.


Here's a comparison done the naive or obvious way. Copy and paste it into an
interactive Python session:


from itertools import takewhile
from timeit import Timer

def stop(): raise StopIteration

setup = 'from __main__ import stop, takewhile'

t1 = Timer('list(i for i in xrange(1000) if i < 50 or stop())', setup)
t2 = Timer('[i for i in takewhile(lambda x: x < 50, xrange(1000))]', setup)

min(t1.repeat(number=100000, repeat=5))
min(t2.repeat(number=100000, repeat=5))


On my computer, t1 is about 1.5 times faster than t2. But this is misleading,
because it's not takewhile that is slow. I am feeding something slow into
takewhile. If I really need to run as fast as possible, I can optimize the
function call inside takewhile:


from operator import lt
from functools import partial

small_enough = partial(lt, 50)
setup2 = 'from __main__ import takewhile, small_enough'

t3 = Timer('[i for i in takewhile(small_enough, xrange(1000))]', setup2)

min(t3.repeat(number=100000, repeat=5))


On my computer, t3 is nearly 13 times faster than t1, and 19 times faster
than t2. Here are the actual times I get, using Python 2.7:


py> min(t1.repeat(number=100000, repeat=5))  # using the StopIteration hack
1.2609241008758545
py> min(t2.repeat(number=100000, repeat=5))  # takewhile and lambda
1.85182785987854
py> min(t3.repeat(number=100000, repeat=5))  # optimized version
0.09847092628479004



-- 
Steven