[Python-ideas] "Iteration stopping" syntax [Was: Is this PEP-able? for X in ListY while conditionZ:]

Shane Green shane at umbrellacode.com
Mon Jul 1 01:44:56 CEST 2013 

On Jun 29, 2013, at 9:53 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> From: Shane Green <shane at umbrellacode.com>
> Sent: Saturday, June 29, 2013 5:10 AM
> 
> 
>> Thanks Andrew.  My knee jerk reaction was to strongly prefer option two, which sounds like–if I understood correctly, and I’m not sure I do–it keeps both comprehensions and expressions.  Rereading your points again, I must admit I didn’t see much to justify the knee jerk reaction.
> 
> Sorry, it's my fault for conflating two independent choices. Let me refactor things:
> 
> Nobody's talking about getting rid of comprehensions. Today, you have the choice to write [comp] instead of list(comp), and that won't change. However, today these differ in that the latter lets you exit early with StopIteration, while the former doesn't. That's what's proposed to change.
> 
> Choice 1 is about the language definition. With this change, there is no remaining difference between genexps except for returning a list vs. an iterator. That means we could simplify things by defining the two concepts together (or defining one in terms of the other). I don't see any downside to doing that.
> 
> Choice 2 is about the CPython implementation. We can reimplement comprehensions as wrappers around genexps, or we can just add a try…except into comprehensions. The former would simplify the compiler and the interpreter, but at a cost of up to 40% for comprehensions. The latter would leave things no simpler than they are today, but also no slower.
> 
> Once put this way, I think the choices are obvious: Simplify the language, don't simplify the implementation.
> 
>> I do commonly use list comprehensions precisely *because* of the performance impact, and can think of a few places the 40% would be problematic.
> 
> Usually that's a premature optimization.

This kind of implies that it’s likely my use of comprehensions was premature, and therefore detracts from the validity of my usage.  I’ve been releasing building management frameworks built using Python since 1.5.2.  When you implement like the BACnet stack, caches, schedulers, property-level access control, etc.–and many things we would not have needed to develop given if we’d needed them now–in Python to run on embedded devices, you learn exactly where your bottlenecks are.  Exhibiting performance benefits similar to map and filter, a well placed loop replacement could sometimes be the difference between needing to migrate some implementation to C or not.  In our processor bound system it is likely the gains were often greeter than 40%, but the bottom line is that comprehensions are sometimes introduced during performance enhancing refactors, and that those examples would be particularly hard hit by a performance loss just like they benefited from the enhancement.  So it’s not a decision that should be made lightly. 

> For anything simple enough that the iteration cost isn't swamped by your real work, the performance usually doesn't matter anyway.
> 
> But "usually" isn't always, and there definitely are real-world cases where it would hurt.
> 
>> Was there a measurable performance difference with approach 2?
> 
> 
> Once I realized that the right place to put the try is just outside the loop… that makes it obvious that there is no per-iteration cost, only a constant cost.

Oh yeah, don’t want to setup the try/catch on every iteration.  


> 
> If you don't raise an exception through a listcomp, that cost is basically running one more opcode and loading a few more bytes into memory. It adds less than 1% for even a trivial comp that loops 10 times, or for a realistic but still simple comp that loops 3 times.

That’s excellent. 
> 
> I'll post actual numbers for local tests and for benchmarks once I get things finished (hopefully Monday).
> 
>> On Jun 28, 2013, at 8:16 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> 
>> 
>> On Jun 28, 2013, at 18:50, Shane Green <shane at umbrellacode.com> wrote:
>>> 
>>> 
>>> Yes, but it only works for generator expressions and not comprehensions.
>>> 
>>> 
>>> This is the point if options #1 and 2: make StopIteration work in comps either (1) by redefining comprehensions in terms of genexps or (2) by fiat. 
>>> 
>>> 
>>> After some research, it turns out that these are equivalent. Replacing any [comprehension] with list(comprehension) is guaranteed by the language (and the CPython implementation) to give you exactly the same value unless (a) something in the comp raises StopIteration, or (b) something in the comp relies on reflective properties (e.g., sys._getframe().f_code.co_flags) that aren't guaranteed anyway.
>>> 
>>> 
>>> So, other than being 4 characters more verbose and 40% slower, there's already an answer for comprehensions.
>>> 
>>> 
>>> And if either of those problems is unacceptable, a patch for #1 or #2 is actually pretty easy. 
>>> 
>>> 
>>> I've got two different proof of concepts: one actually implements the comp as passing the genexp to list, the other just wraps everything after the BUILD_LIST and before the RETURN_VALUE in a the equivalent of try: ... except StopIteration: pass. I need to add some error handling to the C code, and for #2 write sufficient tests that verify that it really does work exactly like #1, but I should have working patches to play with in a couple days.
>>> 
>>> My opinion of that workaround is that it’s also a step backward in terms of readability.  I suspect. 
>>> 
>>>> 
>>>> if i < 50 else stop() would probably also work, since it throws an exception.  That’s better, IMHO.  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Jun 28, 2013, at 6:38 PM, Andrew Carter <acarter at g.hmc.edu> wrote:
>>>> 
>>>> Digging through the archives (with a quick google search) http://mail.python.org/pipermail/python-ideas/2013-January/019051.html, if you really want an expression it seems you can just do
>>>>> 
>>>>> 
>>>>> def stop():
>>>>>   raise StopIteration
>>>>> list(i for i in range(100) if i < 50 or stop())
>>>>> 
>>>>> 
>>>>> it seems to me that this would provide syntax that doesn't require lambdas.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Jun 28, 2013 at 4:50 PM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 28, 2013 at 7:38 PM, Shane Green <shane at umbrellacode.com> wrote:
>>>>>> 
>>>>>> ..
>>>>>>> [x until condition for x in l ...] or 
>>>>>>> [x for x in l until condition]
>>>>>> 
>>>>>> 
>>>>>> Just to throw in one more variation:
>>>>>> 
>>>>>> 
>>>>>> [expr for item in iterable break if condition]
>>>>>> 
>>>>>> 
>>>>>> (inversion of "if" and "break"reinforces the idea that we are dealing with an expression rather than a statement - compare with "a if cond else b")  
>>>>>> _______________________________________________
>>>>>> Python-ideas mailing list
>>>>>> Python-ideas at python.org
>>>>>> http://mail.python.org/mailman/listinfo/python-ideas
>>>> 
>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> http://mail.python.org/mailman/listinfo/python-ideas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130630/0727944c/attachment-0001.html>

More information about the Python-ideas mailing list