[Python-ideas] while conditional in list comprehension ??

[Steven D'Aprano]

On 30/01/13 20:46, Wolfgang Maier wrote:

> b) I have to say I was very impressed by the speed gains you report through the
> use of 'partial', which I had not thought of at all, I have to admit.
> However, I tested your suggestions and I think they both suffer from the same
> mistake:
> your condition is 'partial(lt,50)', but this is not met to begin with and
> results in an empty list at least for me. Have you two actually checked the
> output of the code or have you just timed it? I found that in order to make it
> work the comparison has to be made via 'partial(gt,50)'.

Yes, you are absolutely correct. I screwed that up badly. I can only take comfort
that apparently so did Yuriy.

I don't often paste code in public without testing it, but when I do, it
invariably turns out to be wrong.



> With this modification
> the resulting list in your example would be [0,..,49] as it should be.
>
> And now the big surprise in terms of runtimes:
> partial(lt,50) variant:     1.17  (but incorrect results)
> partial(gt,50) variant:    13.95
> if cond or stop() variant:  9.86

I do not get such large differences. I get these:

py> min(t1.repeat(number=100000, repeat=5))  # cond or stop()
1.2582030296325684
py> min(t2.repeat(number=100000, repeat=5))  # takewhile and lambda
1.9907748699188232
py> min(t3.repeat(number=100000, repeat=5))  # takewhile and partial
1.8741891384124756

with the timers t1, t2, t3 as per my previous email.


> I guess python is just smart enough to recognize that it compares against a
> constant value all the time, and optimizes the code accordingly (after all the
> if clause is a pretty standard thing to use in a comprehension).


No, it is much simpler than that. partial(lt, 50) is equivalent to:

lambda x: lt(50, x)

which is equivalent to 50 < x, *not* x < 50 like I expected.

So the function tests 50 < 0 on the first iteration, which is False, and
takewhile immediately returns, giving you an empty list.

I was surprised that partial was *so much faster* than a regular function. But
it showed me what I expected/wanted to see, and so I didn't question it. A lesson
for us all.


> So the reason for your reported speed-gain is that you actually broke out of the
> comprehension at the very first element instead of going through the first 50!

Correct.



-- 
Steven