Performance of lists vs. list comprehensions

[Wolfram Hinderer]

On 19 Jan., 16:30, Gerald Britton <gerald.brit... at gmail.com> wrote:
> >>> Timer("' '.join([x for x in l])", 'l = map(str,range(10))').timeit()
>
> 2.9967339038848877
>
> >>> Timer("' '.join(x for x in l)", 'l = map(str,range(10))').timeit()
>
> 7.2045478820800781

[...]

> 2. Why should the "pure" list comprehension be slower than the same
> comprehension enclosed in '[...]' ?

Others have already commented on "generator expression" vs. "list
comprehension". I'll try to shed some light on the cause of the
slowness.

For me it's
>>> Timer("' '.join([x for x in l])", 'l = map(str,range(10))').timeit()
0.813948839866498
>>> Timer("' '.join(x for x in l)", 'l = map(str,range(10))').timeit()
2.226825476422391

But wait! I'm on Python 3.1 and the setup statement has to be changed
to make this test meaningful.
>>> Timer("' '.join([x for x in l])", 'l = list(map(str,range(10)))').timeit()
2.5788493369966545
>>> Timer("' '.join(x for x in l)", 'l = list(map(str,range(10)))').timeit()
3.7431774848480472

Much smaller factor now.
But wait! If we want to test list comprehension against generator
comprehension, we should try a function that just consumes the
iterable.

>>> setup = """l = list(map(str,range(10)))
... def f(a):
...     for i in a: pass
... """
>>> Timer("f([x for x in l])", setup).timeit()
3.288511528699928
>>> Timer("f(x for x in l)", setup).timeit()
2.410873798206012

Oops! Iteration over generator expressions is not inherently more
expension than iteration over list comprehensions. But certainly
building a list from a generator expression is more expensive than a
list comprehension?

>>> Timer("[x for x in l]", 'l = list(map(str,range(10)))').timeit()
2.088602950933364
>>> Timer("list(x for x in l)", 'l = list(map(str,range(10)))').timeit()
3.691566805277944

Yes, list building from a generator expression *is* expensive. And
join has to do it, because it has to iterate twice over the iterable
passed in: once for calculating the memory needed for the joined
string, and once more to actually do the join (this is implementation
dependent, of course). If the iterable is a list already, the list
building is not needed.