[Python-ideas] PEP 484: Generating test inputs from type hints

[David MacIver]

(Sorry for the manual thread continuation, I wasn't previously on the
mailing list)

I presume 'support' could and would include a version of given() that
> accessed .__annotations__.


This is on my list of planned features independently of PEP 484 FWIW. It's
on the long list of nice to have but minor things that didn't make it into
1.0.

One of the problems is that I am for the moment somewhat committed to
supporting Python 2.7 as well as Python 3 (I really wish I wasn't), or I
would have probably built Hypothesis on using annotations from the
beginning.

To a degree though it's fortunate that I didn't given PEP 484. It doesn't
look like I would be able to build the full range of features Hypothesis
needs for @given using only type annotations, and 484 suggests the goal is
to move to those being the only valid ones.

The reason for this is that Hypothesis deliberately lets you mix strategies
in with the specifiers used for annotation. So for example you can do
something like:

Palindromes = strategy(str).map(lambda x: x + str(reversed(x)))

This is now a SearchStrategy object that still returns strings, but only
ones that are palindromes.

You can now do something like

@given([Palindromes])
def test_this_is_a_list_of_palindromes(xs):
   # I'm not actually sure what interesting things you can say about a list
of palindromes...

So the point is that you can use a SearchStrategy deep inside the "type"
signatures that Hypothesis uses, despite SearchStrategy not actually being
at all type-like.

(It actually used to be the case that an earlier version of SearchStrategy
allowed you to test whether a value could have come from that strategy,
which would have made them a bit more "type-like", but this proved to be a
bad idea and the work that let me remove it was one of the best decisions I
made in the design of Hypothesis)

So basically I would fully intend for Hypothesis to become a consumer of
types and annotations, but I don't think they can be used as a primary
building block of it.

(Note: I don't consider this a problem with PEP 484, which looks like a
good proposal. Hypothesis's requirements are just weird and specialized and
don't quite match type checking)


> Of course, if every 'test_xyz' were to be
> decorated with the same decorator, then the decorator could be omitted
> and the fuzzing moved into the test runner.


I'd be moderately against this being the primary mode of using Hypothesis,
even without the above caveats. One of the things I like about Hypothesis
right now is that because it just gives you Python functions it can be very
unopinionated about what you're using to run your tests (even if in
practice basically everyone is using unittest or pytest).  However it
should be relatively easy to support alongside once I've done some much
needed refactoring of the core example exploration code.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150331/845b3baf/attachment.html>