[Python-ideas] Optional static typing -- the crossroads

[Dave Halter]

I'm late, but it's also quite hard to read 200 mails in two days
(BlaBlaOverflow) :) I've read most of it. I'm also top-posting. Sorry :)
This mail is going to be way shorter this way.

*EuroPython*
I'm a core-dev of Jedi <https://github.com/davidhalter/jedi> (the
autocompletion library). We were having discussions
<https://github.com/davidhalter/jedi/issues/170> about using annotations
for a longer time, but we couldn't decide on a good approach. This has
changed quite drastically at EuroPython were I realized that Jedi is not
alone in wanting annotations to become a way of specifying types. Most
prominently the some of the pylint guys and Andrey of PyCharm seemed
interested to standardize type annotations. This would have led me to
eventually write a mail here on Python ideas (probably together with those
mentioned), describing our needs. Furthermore there were other interested
parties like Cython and Numba devs, because they might eventually use
annotations to improve their type knowledge.

Conclusion of the conference is that Python's static analysis community
wants type annotations. We want them standardized to be actually able to
use them. A third party library would be useful only if it had the same
opportunities like tulip: Being a third party library with the clear goal
of inclusion in the standard library.

*The current proposal*
While I don't fully support mypy's type system, I think it's a step into
the right direction, but I don't think we need it now (BTW: Mypy's
``Protocol`` class is genius and should be adapted into the stdlib anyway).
In the Appendix below, I've described what I came up with. I think my
solution inferior in capabilities, but way easier to understand and without
clutter. Something like Mypy's `typing` should probably be adopted and
standardized in a separate PEP, possibly for 3.6, once we have a few first
experiences. (Maybe if we start early it could still make 3.5).

I think a resulting PEP of this discussion should contain a deprecation
note for all usages other than type checking in Python 3.5. The current
ambiguous nature of annotations is the fact why no static analysis tool
ever checked them.

One thing you have to remember at this point, *that the two biggest issues
for static analysis are builtins and functions that are never called.
Function annotations would solve both partially.*

*Argument Clinic*
One notable side effect of Guido's proposal would be that Argument Clinic
could use annotations in its signatures. This would be a big benefit for
static analysis. because it would finally reveal the types of builtin
functions (input and output!). I tried to convince Larry Hastings to
implement that, but he refused, citing PEP 3107 which states that "This
work will be left to third-party libraries.".

Argument Clinic in combination with type annotations would be a huge win
for the static analysis community.

*Issues*

By far my biggest concern is the fact that type annotations in CPython
don't have any effect on run-time or compile-time. This is really an issue,
because people from other languages will actually think that this is a type
checker. This could be fixed partially by checking annotations at run-time
(and raising warnings [or exceptions with a command line switch]). This way
annotations wouldn't be without semantic meaning, they would actually be
some kind of pre/post conditions. However, this should also not be the goal
of a current PEP. I just wanted to mention it, so that we can keep it in
mind.

*Appendix (My Proposal)*

My proposal (as discussed and evolved with a few good people at EuroPython)
for containers would look something like this:

    def foo(index: int, x: [float], y: {int: str}) -> (float, str):
        return x[index], y[index]

The "default" containers (set, list, dict and tuple) would just serve as a
way of specifying containers. This makes a lot of things less complicated:

- It's easier to understand (everybody knows builtin types). It also feels
natural to me. The example above covers also almost all variations. A tuple
could be expanded by using the ellipsis: `(int, object, ..., float)`.
- No imports needed. People are more likely to use it, if they don't have
to import typing all the time. This is important for static analysis,
people are only likely to use it if it's easier than writing docstrings
with type information.
- Argument clinic could use the same. The standard library quite often
doesn't accept abstract data types, btw.
- It's what people sometimes use in docstrings: ``@rtype: (float, str)`` or
``:rtype: (float, str)``.

I know this doesn't solve the duck typing issue, but if you look at
real-life Python code bases, there are very few instances of actually
implementing a ``Mapping``, etc. For all of that we should write a separate
proposal (with abstract type classes). My proposal is also missing a Union
type and a few other things, but I'd rather start type annotations really
simple and soon than to wait for good libraries to emerge.
I think the typing module (without my additions) would be
counterproductive, because it's just too complicated too understand. Most
people that are using Python don't know ABCs and would have a hard time
dealing with such optional typing. In the end most people would just ignore
it entirely.

Godspeed!
~ Dave


2014-08-15 1:56 GMT+02:00 Guido van Rossum <guido at python.org>:

> I have read pretty much the entire thread up and down, and I don't think I
> can keep up with responding to every individual piece of feedback. (Also, a
> lot of responses cancel each other out. :-)
>
> I think there are three broad categories of questions to think about next.
>
> (A) Do we even need this?
>
> (B) What syntax to use?
>
> (C) Does/should it support <feature X>?
>
> Taking these in turn:
>
> (A) Do we even need a standard for optional static typing?
>
> Many people have shown either support for the idea, or pointed to some
> other system that addresses the same issue. On the other hand, several
> people have claimed that they don't need it, or that they worry it will
> make Python less useful for them. (However, many of the detractors seem to
> have their own alternative proposal. :-)
>
> In the end I don't think we can ever know for sure -- but my intuition
> tells me that as long as we keep it optional, there is a real demand. In
> any case, if we don't start building something we'll never know whether
> it'll be useful, so I am going to take a leap of faith and continue to
> promote this idea.
>
> I am going to make one additional assumption: the main use cases will be
> linting, IDEs, and doc generation. These all have one thing in common: it
> should be possible to run a program even though it fails to type check.
> Also, adding types to a program should not hinder its performance (nor will
> it help :-).
>
> (B) What syntax should a standard system for optional static typing use?
>
> There are many interesting questions here, but at the highest level there
> are a few choices that constrain the rest of the discussion, and I'd like
> to start with these. I see three or four "families" of approaches, and I
> think the first order is to pick a family.
>
> (1) The mypy family. (http://mypy-lang.org/) This is characterized by its
> use of PEP 3107 function annotations and the constraint that its syntax
> must be valid (current) Python syntax that can be evaluated without errors
> at function definition time. However, mypy also supports collecting
> annotations in separate "stub" files; this is how it handles annotations
> for the stdlib and C extensions. When mypy annotations occur inline (not in
> a stub file) they are used to type check the body of the annotated function
> as well as input for type checking its callers.
>
> (2) The pytypedecl family. (https://github.com/google/pytypedecl) This is
> a custom syntax that can only be used in separate stub files. Because it is
> not constrained by Python's current syntax, its syntax is slightly more
> elegant than mypy.
>
> (3) The PyCharm family. (
> http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html)
> This is a custom syntax that lives entirely in docstrings. There is also a
> way to use stub files with this. (In fact, every viable approach has to
> support some form of stub files, if only to describe signatures for C
> extensions.)
>
> (I suppose we could add a 4th family that puts everything in comments, but
> I don't think anyone is seriously working on such a thing, and I don't see
> any benefits.)
>
> There's also a variant of (1) that Łukasz Langa would like to see -- use
> the syntactic position of function annotations but using a custom syntax
> (e.g. one similar to the pytypedecl syntax) that isn't evaluated at
> function-definition time. This would have to use "from __future__ import
> <something>" for backward compatibility. I'm skeptical about this though;
> it is only slightly more elegant than mypy, and it would open the
> floodgates of unconstrained language design.
>
> So how to choose? I've read passionate attacks and defenses of each
> approach. I've got a feeling that the three projects aren't all that
> different in maturity (all are well beyond the toy stage, none are quite
> ready for prime time). In terms of specific type system features (e.g.
> forward references, generic types, duck typing) I expect they are all
> acceptable, and all probably need some work (and there's no reason to
> assume that work can't be done). All support stubs so you can specify
> signatures for code you can't edit (whether C extension, stdlib or just
> opaque 3rd party code).
>
> To me there is no doubt that (1) is the most Pythonic approach. When we
> discussed PEP 3107 (function annotations) it was always my goal that these
> would eventually be used for type annotations. There was no consensus at
> the time on what the rules for type checking should be, but their syntactic
> position was never in doubt. So we decided to introduce "annotations" in
> Python 3 in the hope that 3rd party experiments would eventually produce
> something satisfactory. Mypy is one such experiment. One of the important
> lessons I draw from mypy is that type annotations are most useful to
> linters, and should (normally) not be used to enforce types at run time.
> They are also not useful for code generation. None of that was obvious when
> we were discussing PEP 3107!
>
> I don't buy the argument that PEP 3107 promises that annotations are
> completely free of inherent semantics. It promises compatibility, and I
> take that very seriously, but I think it is reasonable to eventually
> deprecate other uses of annotations -- there aren't enough significant
> other uses for them to warrant crippling type annotations forever. In the
> meantime, we won't be breaking existing use of annotations -- but they may
> confuse a type checker, whether a stand-alone linter like mypy or built
> into an IDE like PyCharm, and that may serve as an encouragement to look
> for a different solution.
>
> Most of the thornier issues brought up against mypy wouldn't go away if we
> adopted another approach: whether to use concrete or abstract types, the
> use of type variables, how to define type equivalence, the relationship
> between a list of ints and a list of objects, how to spell "something that
> implements the buffer interface", what to do about JSON, binary vs. text
> I/O and the signature of open(), how to check code that uses isinstance(),
> how to shut up the type checker when you know better... The list goes on.
> There will be methods whose type signature can't be spelled (yet). There
> will be code distributed with too narrowly defined types. Some programmers
> will uglify their code to please the type checker.
>
> There are questions about what to do for older versions of Python. I find
> mypy's story here actually pretty good -- the mypy codec may be a hack, but
> so is any other approach. Only the __future__ approach really loses out
> here, because you can't add a new __future__ import to an old version.
>
> So there you have it. I am picking the mypy family and I hope we can start
> focusing on specific improvements to mypy. I also hope that somebody will
> write converters from pytypedecl and PyCharm stubs into mypy stubs, so that
> we can reuse the work already put into stub definitions for those two
> systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's
> syntax (initially in addition to their native syntax, eventually as their
> sole syntax).
>
> PS. I realize I didn't discuss question (C) much. That's intentional -- we
> can now start discussing specific mypy features in separate threads (or in
> this one :-).
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/56fe97fd/attachment-0001.html>