[Python-ideas] Generator unpacking

[Andrew Barnert]

On Feb 12, 2016, at 04:09, Edward Minnix <egregius313 at gmail.com> wrote:
> 
> Hello,
> 
> I am a new programmer and have been using Python for a few months.
> 
> I was experimenting the other day with unpacking (lists, tuples, etc.) And I realized something:
> 
> when you type:
> 
> >>> a, b, *rest = count()
> 
> The interpreter gets caught in an infinite loop which I could not kill without terminating my REPL.
> 
> Would there be a way to add generators to the unpackables, even if it was only in the front?

I think what you're _really_ suggesting here is that there should be a way to unpack iterators lazily, presumably by unpacking them into iterators. And if you could come up with a good intuitive way to make that work, that would be a great suggestion. But I don't know of such a way.

Here's some background:

You already_can_ unpack generators and other iterators:

    >>> c = (i*2 for i in range(5))
    >>> a, b, *rest = c
    >>> rest
    [4. 6, 8]

It works something like this:

    >>> _c = iter(c)
    >>> a = next(_c)
    >>> b = next(_c)
    >>> rest = list(_c)

The problem is that when you give it an infinite iterator, like count(), that last step is attempting to create an infinite list, which takes infinite time (well, it'll raise a MemoryError at some point--but that could take a long time, especially if it drives your system into swap hell along the way).

Even for non-infinite iterators, this is sometimes not what you want, because it means you lose all laziness. For example:

    >>> f = sock.makefile('r')
    >>> a, b, *rest = f

In these last two cases, what you actually want is something like this:

    >>> _c = iter(c)
    >>> a = next(_c)
    >>> b = next(_c)
    >>> rest = _c

That would finish immediately, with rest as a lazy iterator instead of a list.

But is that what you want in the first case? Would it still be what you wanted in "a, b, *rest = [1, 2, 3, 4]"? Many novices would be confused if they wrote that, and then printed out rest and got "<list_iterator at 0x108e49ef0>" instead of "[3, 4]".

Maybe that would be acceptable; people would just have to learn to use list(rest) when that's what they want (the same way they do with map, etc.). But it's probably too late for that, for backward compatibility reasons.

Also, consider "a, *rest, b = [1, 2, 3, 4]". I don't think there's any way that could be done lazily.

Meanwhile, you can always write the expanded version out explicitly. (And you can leave off the first line when you know c is already an iterator.) Or you can use itertools.islice to make it more compact:

    >>> a, b = itertools.islice(c, 2)
    >>> rest = c

If you can come up with intuitive syntax, or an automated rule, or something else, to distinguish the case where you want a list from the case where you want a lazy iterator (and without breaking backward compatibility or over-complicating the language), then we could avoid that. If not, learn to love itertools. :) (That's actually good advise anyway--anyone writing the kind of code that needs infinite lists will get a lot out of digging deep into what itertools can do for you.)