[Python-ideas] loosening the restriction on what types may be unpacked using the ** syntax

[Andrew Barnert]

From: Eric Snow <ericsnowcurrently at gmail.com>

Sent: Wednesday, April 2, 2014 11:54 PM


>T he ** keyword arg unpacking syntax in function calls is useful.
> However, currently there is an explicit check for a mapping type
> (presumably PyMapping_Check).

Actually, I believe there's not really an explicit check. In CPython, ext_do_call (hg.python.org/cpython/file/default/Python/ceval.c#l4463) uses PyDict_Update (https://docs.python.org/3/c-api/dict.html#c.PyDict_Update) to copy the kwargs argument to its empty dict, and PyDict_Update, unlike the Python equivalent dict.update, "doesn’t fall back to the iterating over a sequence of key value pairs if the second argument has no “keys” attribute." In PyPy, the relevant code (in Arguments._combine_starstarargs_wrapped, https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/argument.py#cl-95) is different, but has the same effect (view_as_kwargs only works on mappings). I don't know about other implementations.

However, the language does make this an explicit requirement. Section 3.6.4


> So you can pass an object to dict(),

> but that same object *may* not work for keyword arg unpacking.  For
> example:
> 
>>>>  from collections import OrderedDict
>>>>  def f(**kwargs): return kwargs
> ...
>>>>  kwargs = ((chr(i), i) for i in range(65, 68))
>>>>  f(**kwargs)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: f() argument after ** must be a mapping, not generator
>>>>  f(**OrderedDict(kwargs))
> {'B': 66, 'A': 65, 'C': 67}
> 
> I'd like to relax this restriction so that anything you can pass to
> dict is also valid for keyword arg unpacking.

As the docs (https://docs.python.org/3.4/library/stdtypes.html#dict) explain, "anything you can pass to dict" is actually one of three things:

* 0 or more keyword args
* a mapping, followed by 0 or more keyword args
* an iterable, followed by 0 or more keyword args

I assume you're not suggesting that you should be able to pass keyword args into ** unpacking. So presumably you're just suggesting that ** unpacking should take any iterable, instead of just a mapping, and treat it the same way the dict constructor does. That is:

> If [it] is a mapping object, a dictionary
is created with the same key-value pairs as the mapping object. Otherwise,
[it] must be an iteratorobject.  Each item in
the iterable must itself be an iterator with exactly two objects.  The
first object of each item becomes a key in the new dictionary, and the
second object the corresponding value.  If a key occurs more than once, the
last value for that key becomes the corresponding value in the new
dictionary.

By the way, I think the example would be a lot less confusing if you didn't use a kwargs parameter. While we're at it, it would also be less confusing if you used a list instead of a genexpr; as written, if the first call worked, the second call would get an exhausted iterator and therefore have an empty kwargs. And if you just used a dict instead of an OrderedDict, because that's irrelevant to the issue here.

So, current behavior:

    >>> def f(a, b, c, d): return a, b, c, d
    >>> kwargs = [(chr(i), i) for i in range(65, 68)]
    >>> f(**dict(kwargs))
    (65, 66, 67, 68)
    >>> f(**kwargs)
    Traceback (most recent call last):
       File "<stdin>", line 1, in <module>
     TypeError: f() argument after ** must be a mapping, not generator


Proposed behavior:

    >>> def f(a, b, c, d): return a, b, c, d
    >>> kwargs = [(chr(i), i) for i in range(65, 68)]
    >>> f(**dict(kwargs))
    (65, 66, 67, 68)
    >>> f(**kwargs)
    (65, 66, 67, 68)

This would be a pretty simple change to the language. Section 6.3.4 (https://docs.python.org/3.4/reference/expressions.html#calls) says this:

> If the syntax **expressionappears in the function call, expressionmust
evaluate to a mapping, the contents of which are treated as additional keyword
arguments.  In the case of a keyword appearing in both expressionand as an
explicit keyword argument, a TypeErrorexception is raised.

Change it to this:

> If the syntax **expression appears in the function call, expression must
evaluate to an iterable. If the iterable is a mapping, its contents are treated as additional keyword
arguments. Otherwise, each item in the iterable must itself be an iterator with exactly two objects. The first object of each item is treated as an additional keyword, and the second as that keyword's argument. In the case of a keyword appearing in both expressionand as an
explicit keyword argument, a TypeErrorexception is raised.

As for implementation, that's also trivial. In CPython, just change ext_do_call to try falling back to PyDict_MergeFromSeq2 if PyDict_Update returns a TypeError. In PyPy, just change view_as_kwargs to work on iterables of iterable pairs. And so on.