Python Iterables struggling using map() built-in

[Ivan Evstegneev]

Awesome Ned,

Believe it or not, but I was browsing web for the answer about a half an
hour ago.

Guess what? I found your web page with the explanations you provided there.
))) 

Finally, I was ready to send this question to you directly, cause I didn't
know that you subscribed to this mailing list too. ^_^

But, you was a bit faster.  What a surprise. ))) ))) )))

Thanks a lot for your answer.

Ivan.
  

-----Original Message-----
From: Python-list
[mailto:python-list-bounces+webmailgroups=gmail.com at python.org] On Behalf Of
Ned Batchelder
Sent: Sunday, December 7, 2014 17:29
To: python-list at python.org
Subject: Re: Python Iterables struggling using map() built-in

On 12/6/14 11:44 AM, Ivan Evstegneev wrote:
> And as I've promised the question section:
>
> 1.What actually map() trying to do in Python 3.X?
>
> I mean, why is this works fine:
>
>>>>L = [1, 2, 3, 4]
>
>>>> k = iter(L)
>
>>>> next(k)
>
> 1
>
> and so on.
>
> But not this:
>
>              >>> list(map(iter, L))
>
> Traceback (most recent call last):
>
> File "<pyshell#88>", line 1, in <module>
>
>               print(list(map(iter, L)))
>
> TypeError: 'int' object is not iterable

Let's unpack the code.  You are running:

     map(iter, L)

which is equivalent to:

     map(iter, [1, 2, 3, 4])

which executes:

     iter(1), iter(2), iter(3), iter(4)

If you try iter(1), you get the error you are seeing.  Integers are not
iterable.  What values would it produce?

You ask what this is doing in Python 3, but it happens in any Python
version, because integers are not iterable.

>
> 2.Why strings are allowed "to become" an iterators(self-iterators)?
> Maybe because of files(reading from file) ?
>
> I mean why, is this possible:
>
>>>> print(list(map(iter, S)))
>
> [<str_iterator object at 0x02E24FF0>,
>
> <str_iterator object at 0x02E24CF0>,
>
> <str_iterator object at 0x02E24E10>,
>
> <str_iterator object at 0x02E24DF0>]
>

This is a confusing thing in Python: strings are iterable, they produce a
sequence of 1-character strings:

     >>> list("hello")
     ['h', 'e', 'l', 'l', 'o']

This isn't because of reading from files.  Open files are iterable, they
produce a sequence of strings, one for each line in the file.  This is why
you can do:

     for line in file:
         process(line)

Many times, it would be more convenient if strings were not iterable, but
they are, and you need to keep it in mind when writing general-purpose
iteration.

>
> 3.The last question
>
> Author says:
>
> " /But it falls into an infinite loop and fails in Python 3.X, because
> the 3.X map returns a /
>
> /one-shot iterable object instead of a list as in 2.X. In 3.X, as soon
> as we've run the list /
>
> /comprehension inside the loop once, iters will be exhausted but still
> True/. /To make this /
>
> /work in 3.X, we need to use the list built-in function to create an
> object that can support /
>
> /multiple iterations". /(Like:"Wat?!" ^_^)//
>
> Why the infinite loop would be there and why should list() to make it
> finite?  o_0
>

OK, let's go slowly.  There are a few foundational concepts to get under 
our belt.

*** CONCEPTS

1. An iterable is an object that you can pass to iter() to get an 
iterator.

2. An iterator is an object that can provide you with values one after 
the other, by using next().  next() will either return the next value, 
or raise a StopIteration exception, indicating that there are no more 
values.

3. The only operation supported on iterators is next().  You cannot 
start them over, you cannot ask if there will be more values, you cannot 
find out how many values there will be, you can't ask what the last 
value was, etc.  By supporting only one operation, they allow the 
broadest possible set of implementations.

4. You can ask an iterator for an iterator, and it will return itself. 
That is:  iter(some_iterator) is some_iterator.

5. The "for NAME in EXPR" construct is equivalent to this:

     expr_iter = iter(EXPR)
     try:
         while True:
             NAME = next(expr_iter)
             ..DO_SOMETHING..
     except StopIteration:
         pass

6. In Python 2, map() produces a list of values. Lists are iterable. In 
Python 3, map() produces a map object, which is an iterator.

*** PYTHON 2 EXECUTION

OK, now, here is the code in question:

     1. def myzip(*args):
     2.     iters = map(iter, args)
     3.     while iters:
     4.         res = [next(i) for i in iters]
     5.         yield tuple(res)

Let's cover the Python 2 execution first.  At line 2, map produces a 
list of iterators.  Line 3 will loop forever.  Nothing ever changes the 
list.  In fact, this is a very confusing part of this code.  The code 
should have said "while True" here, because it would work exactly the same.

At line 4, we loop over our list of iterators, and pull the next value 
from each one.  HERE'S THE IMPORTANT PART: because iters is a list, it 
is an iterable, and the for loop on this line will make a new list 
iterator to visit each iterator in turn.  Every time this line is 
executed, every iterator in the list will be next'ed.

Now res is a list of values, and line 5 makes a tuple of them to return.

The loop continues until the next(i) on line 4 fails because one of the 
iterators is emptied, at which point next(i) will raise StopIteration. 
myzip is a generator, so that exception will bubble up to the iteration 
happening outside myzip, and end it.  This is why this code completes on 
Python 2.

*** PYTHON 3 EXECUTION

The difference with Python 3 is that line 2 makes iters be a map 
iterator object, rather than a list.  That means at line 4, the iterator 
is exhausted on the first execution.  The next time around the loop, 
line 4 has no iterators to get from iters, meaning the list 
comprehension there will always produce an empty list.  next(i) isn't 
ever executed on line 4 again, which means it never gets a chance to 
raise StopIteration.  Without line 4 raising StopIteration, the myzip 
generator will never end, producing an infinite loop.

The solution, as the author indicated, is to change line 2 to:

     iters = list(map(iter, args))

This makes the execution on Python 3 the same as on Python 2, where map 
produced a list in the first place.  list() here will iterate over the 
map iterator object, and return a list of all the values (iterators) it 
produced.

NOTE: THIS EXAMPLE IS HORRIBLE.  This code is crazy-confusing, and 
should never have been used as an example of iteration. It layers at 
least three iterations on top of each other, making it very difficult to 
see what is going on.  It uses "while iters" where "while True" would do 
exactly the same thing (iters will never be false).


-- 
Ned Batchelder, http://nedbatchelder.com

-- 
https://mail.python.org/mailman/listinfo/python-list