[Python-ideas] iterable.__unpack__ method

[Nick Coghlan]

On Sat, Feb 23, 2013 at 8:18 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> It would be much easier, and have much the same effect, if unpacking simply
>> requested the minumum number of items and stopped raising a ValueError if
>> the iterator has more items. No new protocol is needed. Guido rejected this
>> as silently masking errors.
>
> What if it were done explicitly, though?
>
> Currently:
>>>> a,b,c=range(3)
>>>> a,b,c=range(4)
> Traceback (most recent call last):
>   File "<pyshell#55>", line 1, in <module>
>     a,b,c=range(4)
> ValueError: too many values to unpack (expected 3)
>>>> a,b,c,*d=range(6)
>>>> a,b,c,* =range(4)
> SyntaxError: invalid syntax
>
> Suppose the last notation were interpreted as "request values for a,
> b, and c, and then ignore the rest".

It highly unlikely it will ever be interpreted that way, because it
contradicts the way the unnamed star is used in function headers (that
is, to indicate the start of the keyword only arguments *without*
accepting an arbitrary number of positional arguments). If you want to
ignore excess values in unpacking, a double-underscore is the best
currently available option (even though it has the downside of not
working well with infinite iterators or large sequences):

>>> a,b,c,*__ =range(4)

However, Alex's original idea of an "unpack protocol" (distinct from,
but falling back to, the ordinary iterable protocol) definitely has at
least a few points in its favour.

1. Iterating over heterogeneous types doesn't really make sense, but
unpacking them does. A separate protocol lets a type support unpacking
assignment without supporting normal iteration.
2. The protocol could potentially be designed to allow an *iterable*
to be assigned to the star target rather than requiring it to be
unpacked into a tuple. This could be used not only to make unpacking
assignment safe to use with infinite iterators, but also to make it
cheaper with large sequences (through appropriate use of
itertools.islice in a custom container's __unpack__ implementation)
and with virtual sequences like range() objects.
3. As Alex noted, if a function that previously returned a 2-tuple
wants to start returning a 3-tuple, that's currently a backwards
incompatible change because it will break unpacking assignment. With
an unpack protocol, such a function can return an object that behaves
like a 3-tuple most of the time, but also transparently supports
unpacking assignment to two targets rather than only supporting three.

I would suggest a different signature for "__unpack__", though, built
around the fact the star target can be used at most once, but in an
arbitrary location:

    def __unpack__(target_len, star_index=None):
        """Unpack values into an iterable of the specified length.

        If star_index is not None, the value at that index will be
        an iterable representing the remaining contents that were not
        unpacked.
        """
        ...

While I'm somewhere between +0 and +1 on the idea myself, there are
some open questions/potential downsides to the idea:

- this change increases the complexity of the language, by explicitly
separating the concept of heterogeneous unpacking from homogeneous
iteration. Those *are* two distinct concepts though, so the increased
complexity may be justifiable.
- for function parameter unpacking, would we allow "*args" to be an
arbitrary iterable rather than promising that it is always a tuple? We
would probably want to do so for consistency with unpacking assignment
under this scheme, which may lead to some confusing error messages
thrown by existing functions which expect it to always be a tuple.
- we would want to document very clearly that it remains possible to
bypass custom unpacking by explicitly calling iter(obj) rather than
using obj directly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia