[Python-Dev] Experiences with Creating PEP 484 Stub Files

[Phil Thompson]

On 9 Feb 2016, at 8:54 pm, Guido van Rossum <guido at python.org> wrote:
> 
> [Just adding to Andrew's response]
> 
> On Tue, Feb 9, 2016 at 9:58 AM, Andrew Barnert via Python-Dev
> <python-dev at python.org> wrote:
>> On Feb 9, 2016, at 03:44, Phil Thompson <phil at riverbankcomputing.com> wrote:
>>> 
>>> There are a number of things I'd like to express but cannot find a way to do so...
>>> 
>>> - objects that implement the buffer protocol
>> 
>> That seems like it should be filed as a bug with the typing repo. Presumably this is just an empty type that registers bytes, bytearray, and memoryview, and third-party classes have to register with it manually?
> 
> Hm, there's no way to talk about these in regular Python code either,
> is there? I think that issue should be resolved first. Probably by
> adding something to collections.abc. And then we can add the
> corresponding name to typing.py. This will take time though (have to
> wait for 3.6) so I'd recommend 'Any' for now (and filing those bugs).

Ok.

>>> - type objects
> 
> You can use 'type' for this (i.e. the builtin). You can't specify any
> properties for types though; that's a feature request:
> https://github.com/python/typing/issues/107 -- but it may be a while
> before we address it (it's not entirely clear how it should work, and
> we have many other pressing issues still).

Yes, I can use type.

>>> - slice objects
> 
>> Can't you just use the concrete types type and slice tor these two? I don't think you need generic or abstract "any metaclass, whether inheriting from type or not" or "any class that meets the slice protocol", do you?
> 
> Can't you use 'slice' (i.e. the builtin)? Mypy supports that.

Yes, I can use slice.

>>> - capsules
>> 
>> That one seems reasonable. But maybe there should just be a types.Capsule Type or types.PyCapsule, and then you can just check that the same as any other concrete type?
>> 
>> But how often do you need to verify that something is a capsule, without knowing that it's the *right* capsule? At runtime, you'd usually use PyCapsule_IsValid, not PyCapsule_CheckExacf, right? So should the type checker be tracking the name too?
>> 
>>> - sequences of fixed size (ie. specified in the same way as Tuple)
> 
> That's kind of a poor data structure. :-( Why can't you use Tuple here?

Because allowing any sequence is more flexible that only allowing a tuple.

>> How would you disambiguate between a sequence of one int and a sequence of 0 or more ints if they're both spelled "Sequence[int]"? That isn't a problem for Tuple, because it's assumed to be heterogeneous, so Tuple[int] can only be a 1-tuple. (This was actually discussed in some depth. I thought it would be a problem, because some types--including tuple itself--are sometimes used as homogenous arbitrary-length containers and sometimes as heterogeneous fixed-length containers, but Guido and others had some good answers for that, even if I can't remember what they were.)
> 
> We solved that by allowing Tuple[int, ...] to spell a homogeneous
> tuple of integers.
> 
>>> - distinguishing between instance and class attributes.
>> 
>> Where? Are you building a protocol that checks the data members of a type for conformance or something? If so, why is an object that has "spam" and "eggs" as instance attributes but "cheese" as a class attribute not usable as an object conforming to the protocol with all three attributes? (Also, does @property count as a class or instance attribute? What about an arbitrary data descriptor? Or a non-data descriptor?)
> 
> It's a known mypy bug. :-( It's somewhat convoluted to fix.
> https://github.com/JukkaL/mypy/issues/1097
> 
> Some things Andrew snipped:
> 
>> The documentation is incomplete - there is no mention of Set or Tuple for example.
> 
> Tuple is here: https://docs.python.org/3/library/typing.html#typing.Tuple

Yes, I missed that.

> collections.Set maps to typing.AbstractSet
> (https://docs.python.org/3/library/typing.html#typing.AbstractSet;
> present twice in the docs somehow :-( ). typing.Set (corresponding to
> builtins.set) is indeed missing, I've a note of that:
> http://bugs.python.org/issue26322.
> 
>> I found the documentation confusing regarding Optional. Intuitively it seems to be the way to specify arguments with default values. However it is explained in terms of (for example) Union[str, None] and I (intuitively but incorrectly) read that as meaning "a str or None" as opposed to "a str or nothing".
> 
> But it *does* mean 'str or None'. The *type* of an argument doesn't
> have any bearing on whether it may be omitted from the argument list
> by the caller -- these are orthogonal concepts (though sadly the word
> optional might apply to both). It's possible (though unusual) to have
> an optional argument that must be a str when given; it's also possible
> to have a mandatory argument that may be a str or None.

In the case of Python wrappers around a C++ library then *every* optional argument will have to have a specific type when given.

So you are saying that a mandatory argument that may be a str or None would be specified as Union[str, None]? But the docs say that that is the underlying implementation of Option[str] - which (to me) means an optional argument that should be a string when given.

> Can you help improve the wording in the docs (preferably by filing an issue)?

When I eventually understand what it means...

>> bytes can be used as shorthand for bytes, bytearray and memoryview - but what about objects that really only support bytes? Shouldn't the shorthand be called something like AnyBytes?
> 
> We debated that, but found it too annoying to have to import and write
> write AnyBytes in so many places. The type checker may not be precise
> for cases that only accept bytes, but hopefully it's more useful in
> general this way.
> 
>> Is there any recommended way to test the validity and completeness of stub files? What's the recommended way to parse them?
> 
> That's also an open issue. For a quick check I tend to just point mypy
> at a stub file, since it is the most mature implementation of PEP 484
> to date (Google's pytype is still working on PEP 484 compatibility).
> While this doesn't always catch all errors, it will at least find
> syntax errors and cases that mypy doesn't support. :-)

Ok I'll try that.

Phil