[Python-ideas] Preserving **kwargs order

[Eric Snow]

On Apr 3, 2014 2:55 AM, "Andrew Barnert" <abarnert at yahoo.com> wrote:
> It's definitely a big deal. In Python 3.4, it is trivial to write a
wrapper that perfectly forwards its arguments. Or that modifies its
arguments in somewhat, then perfectly forwards the modified version. This
is critical to being able to write decorators. Nearly every existing
decorator in the stdlib, third-party libs, ActiveState recipes, and even
your own projects does this. I'll show a dead-simple example below.
>
>
> But with either of your options, that would no longer be true. It would
be at least a little more difficult to write a wrapper that perfectly
forwards its arguments, and every existing decorator in the world would
have to be rewritten to do so.

I'll concede "a little more difficult" in the pass-through case (with
unaware decorators being the main issue).  :)  One reason why I don't think
it's a huge problem in the decorator case is that you decorate your own
functions (though not necessarily with your own decorators).  So if you
change an existing function to use ordered kwargs, then you would already
be focused on the feature.  You would ensure any decorators you've applied
to the function accommodate ordered pass-through, perhaps adapting/wrapping
the decorators to do so.  It's all pretty localized in your code, so it
should be hard to miss.

I agree that both of the approaches I've mentioned require this extra work.
 I also agree that it would be good to avoid this complication.  I have a
few more ideas which I'll include in the upcoming PEP (and I've summarized
below).

Other than the pass-through case, the 2 outstanding approaches seem fine.
 Existing functions would probably need a signature change anyway and for
new functions no one has to change any code. :)  Either way, if a function
depends on ordered kwargs, then that's the sort of important feature that
you will make clear to people using your function, on par with the rest of
the signature.

[snipped example]

> Can Python fix this for you magically?

I'll drop this proposal in a heartbeat if it requires something like that!
 I just don't think it does, even with the 2 less-optimal solutions we've
been discussing.  (My definition of magic may be different than yours
though. <wink>)

>
> The only way to make eggs end up as order-preserving is to change Python
so that all functions (or at least all functions defined inside other
functions) are order-preserving.
>
> Either you or someone else suggested that if Python could just pass the
kwargs dict straight through instead of unpacking and repacking it, that
would help. But it wouldn't.

Agreed.  (Pretty sure that wasn't me. :) )  That would be a non-option for
other reasons anyway.

[more snipped]
> Or consider your original example (with explicit a and b params on spam
before **kwargs) with memoize, where the kwargs in eggs has a, b, c, and d,
but the one inside spam has only c and d.

That should be a non-issue given how the data is packed into a new kwargs.

>
> Now, obviously you could fix any particular wrapper. Presumably your
magic decorator works by doing something detectable from Python, like a new
co_flags flag. So, we could define two new functions (presumably in inspect
and functools, respectively):
>
>
>     def preserves_kwargs_order(func):
>         return func.__code__.co_flags & 16
>
>     def wrap_preserve_kwargs_order(func):
>         def wrapper(inner):
>             if inspect.preserves_kwargs_order(func):
>                 return preserve_kwargs_order(inner)
>             else:
>                 return inner
>         return wrapper
>
> And now, all you have to do is add one line to most decorators and
they're once again perfect forwarders:
>
>     def memoize(func):
>
>         _cache = {}
>         @functools.wrap_preserve_kwargs_order(func)
>         def wrapper(*args, **kwargs):
>             if (args, kwargs) not in _cache:
>                 _cache[args, kwargs] = func(*args, **kwargs)
>             return _cache[args, kwargs]
>         return wrapper
>
>
> But the point is, you still have to add that decorator to every wrapper
function in the world or they're no longer perfect forwarders.

Nice.

>
> You could even add that wrap_preserve_kwargs_order call into
functools.wraps, and that would fix _many_ decorators (because many
decorators are written to use functools.wraps), but still not all.

Also nice.

That said, I agree that it would be best to come up with a solution that
drops in without breaking the "perfect forwarders", thus rendering those
tools unnecessary.  Here are 3 ideas I had last night and today:

1. Add a '__order__' attribute to dict that defaults to None, and always
set it for kwargs (to a frozen list/tuple).
2. Use a minimal dict subclass just for kwargs that provides __order__
(meaning all other uses of dict don't take the memory hit).
3. Use OrderedDict for kwargs by default, and provide a decorator that
people can use to get just a dict (a.k.a. the current behavior).  It's only
a few extreme cases where OrderedDict is problematic.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140403/63b5b5f7/attachment-0001.html>