[Python-ideas] Proposal: Use mypy syntax for function annotations

[Yann Kaiser]

I'd like to offer my perspective(and objections) on this:

I author and maintain a reflective argument parser for python, clize[1],
possibly in the same vein as Ethan's. While it originally did not support
any Python 3 features, it has come to accept keyword-only parameters and
annotations for alternate parameter names and conversion functions(which
may look like type annotations to a reader looking for them). The upcoming
version will completely deprecate the old "pass a bunch of unsightly
arguments to a decorators[2]" API, and use annotations exclusively[3],
using backports[4][5] to make them and keyword-only parameters available to
Python 2 users. Clize has a handful of users, with 100 stars on GitHub, and
~75 downloads per day according to PyPI[1].

Needless to say, the idea of deprecating uses of parameter and return
annotations that aren't this particular way of typechecking is upsetting to
me.

On to the most formal of formal complaints: PEP 3107[6] rejected
standardizing typing-related annotations. What has changed since then, or
what has otherwise invalidated that conclusion, that it should be
backtracked on and changed? GvR says he knows of little practical use of
function annotations in mainstream code. Here's the (unfortunate) reason:
Mainstream code is either incompatible with Python 3 or maintaining
compatibility between both Python 2 and Python 3. Standard library code
which had no such concern for the most part stuck with what stood in PEP
3107, which was not to teach annotations as typing info. Short of resorting
to tools like sigtools.modifiers.annotate[4], *mainstream code can't use
function annotations*.

You will argue that people are already coming up with type-checking
libraries en masse, and that little alternative uses have come up. That's
because type-checking is an idea that people have seen and had in mind well
before now, PEP 3107, or Python itself. It is even one of the two examples
in the PEP, the other being fairly irrelevant given docstrings, IMO.
Mainstream code can't use annotations. New brains are mostly taught to use
python 2[7] or keep compatibility with it, so no new ideas there(save for
the endless stream of "I've used Python for a day and it needs static
typing" coming from statically-typed languages.) IMO there simply hasn't
been enough time for new uses to show up. There's the argument parsers me
and a couple others have mentioned, and although the idea can be ported to
other interfaces(web forms, GUI forms?), this is only one use yet, but I
think it is a good one, one that is also fairly unique to Python. Do we
want to close this window of opportunity by trying to imitate the 80's?

Even if you don't deprecate other uses, putting this in the standard
library sends a message to the mob: "Annotations are meant to express types
and other uses should conform." It's already slightly against-current to
propose reflective tools because "reflection is weird and unreliable," I'd
hate to see where this would take us.

Worse than shutting down a potential area of innovation, promoting
type-checking in such a way would alienate what many experienced Python
programmers view as a core tenet of Python: duck-typing.

Ironically, GvR's example illustrates it perfectly:

  from typing import List, Dict

  def word_count(input: List[str]) -> Dict[str, int]:
      result = {}  #type: Dict[str, int]
      for line in input:
          for word in line.split():
              result[word] = result.get(word, 0) + 1
      return result

First, there's the obvious mistake others have noted: this thing does not
need a list, it needs an iterable, and that iterable merely needs to yield
objects that have a split method, which yields... hashable objects. That's
it. This function could work just as well if line.split yielded integers.
It's what's awesome about things like collections.Counter. You can give it
whatever iterable you like, whether it yields characters, strings,
integers, the types of objects tracked by the garbage collector, you name
it. This is awesome. Declaring and casting between type templates is
confusing, verbose, scary and not awesome in general. It just makes me
think of C++ (sorry).

Let's say that this function needs to operate on strings, because at some
point in the future it will replace line.split() with something that
accounts for punctuation and whatnot, and ignore that this would really
mean it actually became a different function. (Do you expect str.split to
begin handling punctuation overnight?) Now the only mistake is that it
pretends it is specific to lists. This mistake is easily realized, so what
happens when you write your fancy library that fully declares its typing,
declaring generic interfaces where appropriate, and one of your
dependencies finally makes the switch over and gets it wrong? Now your
typechecker says your functions are making incorrect calls to that API
because you passed the result of a generator function to something that
expects a list. How often do you think this will happen? How many people
eager to type-check everything will not know how duck-typing matters in
their library or program and specify overly restrictive interfaces for
their functions? How many will assume checking declared types is a
substitute for unit testing? This is speculative, but being a regular
participant of #python on FreeNode, I already dread this.

Finally, if I abstract this proposal, it boils down to: The proposal wants
to take what has been so far a fully free-for-all attribute of functions,
and make it free-for-all-but-only-for-this-purpose. That's a bit... weird?
Maybe the mypy project could split its declaration and checking components,
sort of like I split clize(CLI argument parsing) and sigtools(high-level
signature operations and reflection improvements)? Isn't the way those
libraries declare types-to-be-checked the main way they will distinguish
each other for developers? Is there a problem preventing IDEs from
implementing a runner for mypy, much like eg. the way you can use pyflakes
as your compiler in vim?

Sorry for the long rant.

[1] Clize on PyPI: https://pypi.python.org/pypi/clize
[2] Old clize API:
https://github.com/epsy/clize/blob/e84637a631574e793719114ad7e40d0b36df1a78/README.rst#using-clize-in-your-programs
[3] New clize API involving annotations:
http://clize.readthedocs.org/en/latest/basics.html#converting-arguments
[4] Inspect.signature-compatible backport of annotations:
http://sigtools.readthedocs.org/en/latest/#sigtools.modifiers.annotate
[5] Backport of inspect.signature: https://pypi.python.org/pypi/funcsigs/0.4
[6] Rejected proposals from Function Annotations PEP:
http://legacy.python.org/dev/peps/pep-3107/#rejected-proposals
[7] Warnings for Beginners - Learn Python the Hard Way:
http://learnpythonthehardway.org/book/ex0.html#warnings-for-beginners

-Yann Kaiser


On 14 August 2014 13:36, Andrey Vlasovskikh <andrey.vlasovskikh at gmail.com>
wrote:

> On Thu, Aug 14, 2014 at 2:28 PM, Manuel Cerón <ceronman at gmail.com> wrote:
>
> > One interesting feature of TypeScript is that it allows you to annotate
> > existing code without modifying it, by using external definition files.
> In
> > the JavaScript world, many people have contributed TypeScript annotation
> > files for popular JS libraries (http://definitelytyped.org/).
> >
> > I think this is possible in Python as well doing something like this:
> >
> > @annotate('math.ciel')
> > def ciel(x: float) -> int:
> >     pass
> >
> > I think this should be the way to go for annotating the stdlib. It has
> the
> > advantage that if the type syntax changes, it's possible to provide new
> type
> > annotations without changing the libraries at all, and even supporting
> older
> > versions. In this way the code and type annotations can evolve
> separately.
> >
> > Does mypy support something like this?
>
> We use something quite similar to TypeScript's repository of
> annotations in PyCharm. Here is our external annotations proposal and
> stubs for some stdlib modules
> (https://github.com/JetBrains/python-skeletons) that uses a custom
> type syntax in docstrings due to lack of a better standard, see
> README. We state in our proposal that we would like the standard to
> emerge. The idea being discussed here about using Mypy's type system
> is not new to us. As I've mentioned in the original thread, we have
> discussed it with Jukka Lehtosalo, the author of Mypy. Some initial
> ideas are listed here (https://github.com/pytypes/pytypes).
>
> --
> Andrey Vlasovskikh
> Web: http://pirx.ru/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/d4637c92/attachment.html>