[Python-ideas] PEP 484 change proposal: Allowing @overload outside stub files

[Guido van Rossum]

On Fri, Jan 22, 2016 at 8:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 23 January 2016 at 06:00, Guido van Rossum <guido at python.org> wrote:
> > Ben Darnell (Tornado lead) brought up a good use case for allowing
> @overload
> > in regular Python files.
> >
> > There's some discussion (some old, some new) here:
> > https://github.com/ambv/typehinting/issues/72
> >
> > I now propose to allow @overload in non-stub (i.e. .py) files, but with
> the
> > following rule: a series of @overload-decorated functions must be
> followed
> > by an implementation function that's not @overload-decorated. Calling an
> > @overload-decorated function is still an error (I propose
> NotImplemented).
> > Due to the way repeated function definitions with the same name replace
> each
> > other, leaving only the last one active, this should work. E.g. for
> > Tornado's utf8() the full definition would look like this:
> >
> > @overload
> > def utf8(value: None) -> None: ...
> > @overload
> > def utf8(value: bytes) -> bytes: ...
> > @overload
> > def utf8(value: str) -> bytes: ...  # or (unicode)->bytes, in PY2
> > def utf8(value):
> >     # Real implementation goes here.
>
> I share Andrew's concerns about the lack of integration between this
> and functools.singledispatch, so would it be feasible to apply the
> "magic comments" approach here, similar to the workarounds for
> variable annotations and Py2 compatible function annotations?
>
> That is, would it be possible to use a notation like the following?:
>
>     def utf8(value):
>         # type: (None) -> None
>         # type: (bytes) -> bytes
>         # type: (unicode) -> bytes
>         ...
>
> You're already going to have to allow this for single lines to handle
> Py2 compatible annotations, so it seems reasonable to also extend it
> to handle overloading while you're still figuring out a native syntax
> for that.
>

That's clever. I'm sure we could make it work if we wanted to. But it
doesn't map to anything else -- in stub files do already have @overload,
and there's no way to translate this directly to Python 3 in-line
annotations.

Regarding the confusion with @functools.singledispatch, hopefully all
documentation for @overload (including StackOverflow :-) would quickly
point out how you are supposed to use it.

There's also a deep distinction between @overload in PEP 484 and
singledispatch, multidispatch or even the (ultimately deferred) approach
from PEP 3124, also called @overload.

PEP 484's @overload (whether in stubs or in this proposed form in .py
files) talks to the *type checker* and it can be used with generic types.
For example, suppose you have

@overload
def foo(a: Sequence[int]) -> int: ...
@overload
def foo(a: Sequence[str]) -> float: ...
def foo(a):
    return sum(float(x) for x in a)

(NOTE: Don't be fooled to think that the implementation is the last word on
the supported types and hence the list of overloads is "obviously"
incomplete. The type checker needs to take the overloads at their word and
reject calls to e.g. foo([3.14]). A future implementation that matches the
overloaded signatures given here might not work for float arguments.)

Here the implementation will have to somehow figure out whether its
argument is a list of integers or strings, e.g. by checking the type of the
first item -- that should be okay since passing the type check implies a
promise that the argument is homogeneous. But a purely runtime dispatcher
would not be able to make that distinction so easily, since PEP 484 assumes
type erasure -- at runtime the argument is just a Sequence. Of course,
functools.singledispatch sidesteps this by not supporting generic types at
all. But the example illustrates that the two are more different than you'd
think from the utf8() example (which just distinguishes between unicode,
bytes and None -- no generic types there).

>From a type-checking perspective, functools.singledispatch is not easy to
handle -- it is defined in terms of its runtime behavior, and it explicitly
supports dynamic registration. (Who uses it? There's only one use in the
stdlib, which is in pkgutil.py, under the guise @simplegeneric.)

Clearly both @overload and functools.singledispatch are stepping stones on
the way to an elusive better solution. Hopefully until that solution is
found they can live together?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160122/721eae92/attachment-0001.html>