For-each behavior while modifying a collection

[Ned Batchelder]

On 11/28/13 5:14 PM, Valentin Zahnd wrote:
> 2013/11/28 Ned Batchelder <ned at nedbatchelder.com>:
>> On 11/28/13 10:49 AM, Valentin Zahnd wrote:
>>>
>>> Hello
>>>
>>> For-each does not iterate ober all entries of collection, if one
>>> removes elements during the iteration.
>>>
>>> Example code:
>>>
>>> def keepByValue(self, key=None, value=[]):
>>>       for row in self.flows:
>>>           if not row[key] in value:
>>>               self.flows.remove(row)
>>>
>>> It is clear why it behaves on that way. Every time one removes an
>>> element, the length of the colleciton decreases by one while the
>>> counter of the for each statement is not.
>>> The questions are:
>>> 1. Why does the interprete not uses a copy of the collection to
>>> iterate over it? Are there performance reasons?
>>
>>
>> Because implicit copying would be pointless in most cases.  Most loops don't
>> even want to modify the collection, why copy all iterables just in case your
>> loop might be one of the tiny few that might change the collection?
>>
>
> Okay, I get this point.
> But how is the for-each parsed. Is it realised with a iterator or ist
> done with a for loop where a counter is incremented whilst it is less
> than the length of the list and reads the element at the index of the
> counter?

The for loop creates an iterator over the expression, and pulls values 
from the iterator, running the body once for each value.

This presentation explains the mechanics of loops, along with some 
examples of more exotic uses, and (at least) one bad joke: 
http://nedbatchelder.com/text/iter.html

>
>> Of course, if that prices is acceptable to you, you could do the copy
>> yourself:
>>
>>      for row in list(self.flows):
>>          if row[key] not in value:
>>              self.flows.remove(row)
>>
>>
>>> 2. Why is the counter for the iteration not modified?
>>
>>
>> Because the list and the iterator over the list are different objects. I
>> suppose the list and the iterator could have been written to update when the
>> list is modified, but it could get pretty complicated, even more so if you
>> want to do the same for other collections like dictionaries.
>>
>> The best advice is: don't modify the list, instead make a new list:
>>
>>      self.flows = [r for r in self.flows if r[key] not in value]
>>
> How is the list comprehension done by the interpreter?
> The list I'm have to work with is not that small. So to avoid
> duplicated parts in memory the function looks currently like this:
>
>       def keepByValue(self, key=None, value=[]):
>           tmpFlows = []
>           while len(self.flows) > 0:
>               row = self.flows.pop()
>               if row[key] in value:
>                   tmpFlows.append(row)
>           self.flows = tmpFlows
>
> If there is no duplication in memory, the list comprehension would be
> much more elegant.
>

The list comprehension won't copy the elements of the list, if that is 
what you're worried about.  And it won't reverse the order of your list 
the way this loop does.  BTW, my list comprehension got the sense of the 
condition wrong, it should be:

     self.flows = [r for r in self.flows if r[key] in value]

--Ned.

>> Be careful though, since there might be other references to the list, and
>> now you have two.
>>
>> --Ned.
>>
>>>
>>> Valentin
>>>
>>
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list