Why generators take long time?

[Steven D'Aprano]

On Tue, 19 Jan 2016 06:27 pm, Arshpreet Singh wrote:

> 
> I was playing with Generators and found that using Generators time is bit
> more than list-comprehensions or I am doing it wrong?

Generators may have slightly more overhead than iterating over a list. They
save memory, not time.

On my machine, I see a small difference of about 15% between summing a list
comp or a generator expression:

[steve at ando ~]$ python -m timeit "sum([i for i in xrange(1000)])"
10000 loops, best of 3: 102 usec per loop

[steve at ando ~]$ python -m timeit "sum(i for i in xrange(1000))"
10000 loops, best of 3: 117 usec per loop



But both are MUCH slower than summing the xrange object itself:

[steve at ando ~]$ python -m timeit "sum(xrange(1000))"
10000 loops, best of 3: 21.4 usec per loop



The generator has to pause execution of the code, then start it up again,
over and over again, while the list comprehension can run without
interruption, then iterate over that list.

This is, I think, the fastest way to iterate over the objects, which
eliminates as much of the overhead as possible:


[steve at ando ~]$ python -m timeit -s "from collections import deque" 
    -s "it = iter([i for i in xrange(1000)])" "deque(it, maxlen=0)"
1000000 loops, best of 3: 0.913 usec per loop


[steve at ando ~]$ python -m timeit -s "from collections import deque" 
    -s "it = (i for i in xrange(1000))" "deque(it, maxlen=0)"
1000000 loops, best of 3: 0.965 usec per loop


which is about a 6% difference. So if you eliminate everything else, and
just iterate over a pre-populated list as fast as possible, versus a simple
generator expression, the list is about 6% faster.

BUT there is a lot of variation in these timings. For the list comprehension
version, I ran it four times and got these four results:

1000000 loops, best of 3: 0.913 usec per loop
1000000 loops, best of 3: 0.911 usec per loop
1000000 loops, best of 3: 0.977 usec per loop
1000000 loops, best of 3: 0.938 usec per loop

and for the generator expression:

1000000 loops, best of 3: 0.965 usec per loop
1000000 loops, best of 3: 0.914 usec per loop
1000000 loops, best of 3: 0.981 usec per loop
1000000 loops, best of 3: 0.931 usec per loop

So there's plenty of random variation in the times.




-- 
Steven