[Python-ideas] Pre-conditions and post-conditions
Marko Ristin-Kaufmann
marko.ristin at gmail.com
Sat Sep 22 04:30:23 EDT 2018
Hi,
I implemented a sphinx extension to include contracts in the documentation:
https://github.com/Parquery/sphinx-icontract
The extension supports inheritance. It lists all the postconditions and
invariants including the inherited one. The preconditions are grouped by
classes with ":requires:" and ":requires else:".
I was unable to get the syntax highlighting for in-line code to work --
does anybody know how to do that in Sphinx?
The results can be seen, *e.g.* in this documentation:
https://pypackagery.readthedocs.io/en/latest/packagery.html
On a more general note: is there any blocker left why you would *not *use
the contracts in your code? Anything I could improve or fix in icontract
that would make it more convincing to use (apart from implementing static
contract checking and automatic test generation :))?
Cheers,
Marko
On Thu, 20 Sep 2018 at 22:52, Marko Ristin-Kaufmann <marko.ristin at gmail.com>
wrote:
> Hi,
> Again a brief update.
>
> * icontract supports now static and class methods (thanks to my colleague
> Adam Radomski) which came very handy when defining a group of functions as
> an interface *via* an abstract (stateless) class. The implementors then
> need to all satisfy the contracts without needing to re-write them. You
> could implement the same behavior with *_impl or _* ("protected") methods
> where public methods would add the contracts as asserts, but we find the
> contracts-as-decorators more elegant (N functions instead of 2*N; see the
> snippet below).
>
> * We implemented a linter to statically check that the contract arguments
> are defined correctly. It is available as a separate Pypi package
> pyicontract-lint (https://github.com/Parquery/pyicontract-lint/). Next
> step will be to use asteroid to infer that the return type of the condition
> function is boolean. Does it make sense to include PEX in the release on
> github?
>
> * We plan to implement a sphinx plugin so that contracts can be readily
> visible in the documentation. Is there any guideline or standard/preferred
> approach how you would expect this plugin to be implemented? My colleagues
> and I don't have any experience with sphinx plugins, so any guidance is
> very welcome.
>
> class Component(abc.ABC, icontract.DBC):
> """Initialize a single component."""
>
> @staticmethod
> @abc.abstractmethod
> def user() -> str:
> """
> Get the user name.
>
> :return: user which executes this component.
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: result in groups())
> def primary_group() -> str:
> """
> Get the primary group.
>
> :return: primary group of this component
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: result.issubset(groups()))
> def secondary_groups() -> Set[str]:
> """
> Get the secondary groups.
>
> :return: list of secondary groups
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: all(not pth.is_absolute() for pth in result))
> def bin_paths(config: mapried.config.Config) -> List[pathlib.Path]:
> """
> Get list of binary paths used by this component.
>
> :param config: of the instance
> :return: list of paths to binaries used by this component
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: all(not pth.is_absolute() for pth in result))
> def py_paths(config: mapried.config.Config) -> List[pathlib.Path]:
> """
> Get list of py paths used by this component.
>
> :param config: of the instance
> :return: list of paths to python executables used by this component
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: all(not pth.is_absolute() for pth in result))
> def dirs(config: mapried.config.Config) -> List[pathlib.Path]:
> """
> Get directories used by this component.
>
> :param config: of the instance
> :return: list of paths to directories used by this component
> """
> pass
>
>
> On Sat, 15 Sep 2018 at 22:14, Marko Ristin-Kaufmann <
> marko.ristin at gmail.com> wrote:
>
>> Hi David Maertz and Michael Lee,
>>
>> Thank you for raising the points. Please let me respond to your comments
>> in separation. Please let me know if I missed or misunderstood anything.
>>
>> *Assertions versus contracts.* David wrote:
>>
>>> I'm afraid that in reading the examples provided it is difficulties for
>>> me not simply to think that EVERY SINGLE ONE of them would be FAR easier to
>>> read if it were an `assert` instead.
>>>
>>
>> I think there are two misunderstandings on the role of the contracts.
>> First, they are part of the function signature, and not of the
>> implementation. In contrast, the assertions are part of the implementation
>> and are completely obscured in the signature. To see the contracts of a
>> function or a class written as assertions, you need to visually inspect the
>> implementation. The contracts are instead engraved in the signature and
>> immediately visible. For example, you can test the distinction by pressing
>> Ctrl + q in Pycharm.
>>
>> Second, assertions are only suitable for preconditions. Postconditions
>> are practically unmaintainable as assertions as soon as you have multiple
>> early returns in a function. The invariants implemented as assertions are
>> always unmaintainable in practice (except for very, very small classes) --
>> you need to inspect each function of the class and all their return
>> statements and manually add assertions for each invariant. Removing or
>> changing invariants manually is totally impractical in my view.
>>
>> *Efficiency and Evidency. *David wrote:
>>
>>> The API of the library is a bit noisy, but I think the obstacle it's
>>> more in the higher level design for me. Adding many layers of expensive
>>> runtime checks and many lines of code in order to assure simple predicates
>>> that a glance at the code or unit tests would do better seems wasteful.
>>
>>
>> I'm not very sure what you mean by expensive runtime checks -- every
>> single contract can be disabled at any point. Once a contract is disabled,
>> there is literally no runtime computational cost incurred. The complexity
>> of a contract during testing is also exactly the same as if you wrote it in
>> the unit test. There is a constant overhead due to the extra function call
>> to check the condition, but there's no more time complexity to it. The
>> overhead of an additional function call is negligible in most practical
>> test cases.
>>
>> When you say "a glance at the code", this implies to me that you
>> referring to your own code and not to legacy code. In my experience, even
>> simple predicates are often not obvious to see in other people's code as
>> one might think (*e.g. *I had to struggle with even most simple ones
>> like whether the result ends in a newline or not -- often having to
>> actually run the code to check experimentally what happens with different
>> inputs). Postconditions prove very useful in such situations: they let us
>> know that whenever a function returns, the result must satisfy its
>> postconditions. They are formal and obvious to read in the function
>> signature, and hence spare us the need to parse the function's
>> implementation or run it.
>>
>> Contracts in the unit tests.
>>
>>> The API of the library is a bit noisy, but I think the obstacle it's
>>> more in the higher level design for me. Adding many layers of expensive
>>> runtime checks and many lines of code in order to assure simple predicates
>>> that a glance at the code or *unit tests would do better* seems
>>> wasteful.
>>>
>> (emphasis mine)
>>
>> Defining contracts in a unit test is, as I already mentioned in my
>> previous message, problematic due to two reasons. First, the contract
>> resides in a place far away from the function definition which might make
>> it hard to find and maintain. Second, defining the contract in the unit
>> test makes it impossible to put the contract in the production or test it
>> in a call from a different function. In contrast, introducing the contract
>> as a decorator works perfectly fine in all the three above-mentioned cases
>> (smoke unit test, production, deeper testing).
>>
>> *Library. *Michael wrote:
>>
>>> I just want to point out that you don't need permission from anybody to
>>> start a library. I think developing and popularizing a contracts library is
>>> a reasonable goal -- but that's something you can start doing at any time
>>> without waiting for consensus.
>>
>>
>> As a matter of fact, I already implemented the library which covers most
>> of the design-by-contract including the inheritance of the contracts. (The
>> only missing parts are retrieval of "old" values in postconditions and loop
>> invariants.) It's published on pypi as "icontract" package (the website is
>> https://github.com/Parquery/icontract/). I'd like to gauge the interest
>> before I/we even try to make a proposal to make it into the standard
>> library.
>>
>> The discussions in this thread are an immense help for me to crystallize
>> the points that would need to be addressed explicitly in such a proposal.
>> If the proposal never comes about, it would at least flow into the
>> documentation of the library and help me identify and explain better the
>> important points.
>>
>> *Observation of contracts. *Michael wrote:
>>
>>> Your contracts are only checked when the function is evaluated, so you'd
>>> still need to write that unit test that confirms the function actually
>>> observes the contract. I don't think you necessarily get to reduce the
>>> number of tests you'd need to write.
>>
>>
>> Assuming that a contracts library is working correctly, there is no need
>> to test whether a contract is observed or not -- you assume it is. The same
>> applies to any testing library -- otherwise, you would have to test the
>> tester, and so on *ad infinitum.*
>>
>> You still need to evaluate the function during testing, of course. But
>> you don't need to document the contracts in your tests nor check that the
>> postconditions are enforced -- you assume that they hold. For example, if
>> you introduce a postcondition that the result of a function ends in a
>> newline, there is no point of making a unit test, passing it some value and
>> then checking that the result value ends in a newline in the test.
>> Normally, it is sufficient to smoke-test the function. For example, you
>> write a smoke unit test that gives a range of inputs to the function by
>> using hypothesis library and let the postconditions be automatically
>> checked. You can view each postcondition as an additional test case in this
>> scenario -- but one that is also embedded in the function signature and
>> also applicable in production.
>>
>> Not all tests can be written like this, of course. Dealing with a complex
>> function involves writing testing logic which is too complex to fit in
>> postconditions. Contracts are not a panacea, but they absolute us from
>> implementing trivial testing logic while keeping the important bits of the
>> documentation close to the function and allowing for deeper tests.
>>
>> *Accurate contracts. *Michael wrote:
>>
>>> There's also no guarantee that your contracts will necessarily be
>>> *accurate*. It's entirely possible that your preconditions/postconditions
>>> might hold for every test case you can think of, but end up failing when
>>> running in production due to some edge case that you missed.
>>>
>>
>> Unfortunately, there is no practical exit from this dilemma -- and it
>> applies all the same for the tests. Who guarantees that the testing logic
>> of the unit tests are correct? Unless you can formally prove that the code
>> does what it should, there is no way around it. Whether you write contracts
>> in the tests or in the decorators, it makes no difference to accuracy.
>>
>> If you missed to test an edge case, well, you missed it :). The
>> design-by-contract does not make the code bug-free, but makes the bugs *much
>> less likely* and *easier *to detect *early*. In practice, if there is a
>> complex contract, I encapsulate its complex parts in separate functions
>> (often with their own contracts), test these functions in separation and
>> then, once the tests pass and I'm confident about their correctness, put
>> them into contracts.
>>
>> (And if you decide to disable those pre/post conditions to avoid the
>>> efficiency hit, you're back to square zero.)
>>>
>>
>> In practice, we at Parquery AG let the critical contracts to run in
>> production to ensure that the program blows up before it exercises
>> undefined behavior in a critical situation. The informative violation
>> errors of the icontract library help us to trace the bugs more easily since
>> the relevant values are part of the error log.
>>
>> However, if some of the contracts are too inefficient to check in
>> production, alas you have to turn them off and they can't be checked since
>> they are inefficient. This seems like a tautology to me -- could you please
>> clarify a bit what you meant? If a check is critical and inefficient at the
>> same time then your problem is unsolvable (or at least ill-defined);
>> contracts as well as any other approach can not solve it.
>>
>> *Ergonimical assertions. *Michael wrote:
>>
>>> Or I guess to put it another way -- it seems what all of these contract
>>> libraries are doing is basically adding syntax to try and make adding
>>> asserts in various places more ergonomic, and not much else. I agree those
>>> kinds of libraries can be useful, but I don't think they're necessarily
>>> useful enough to be part of the standard library or to be a technique
>>> Python programmers should automatically use by default.
>>
>>
>> From the point of view of the *behavior, *that is exactly the case. The
>> contracts (*e.g. *as function decorators) make postconditions and
>> invariants possible in practice. As I already noted above, postconditions
>> are very hard and invariants almost impossible to maintain manually without
>> the contracts. This is even more so when contracts are inherited in a class
>> hierarchy.
>>
>> Please do not underestimate another aspect of the contracts, namely the
>> value of contracts as verifiable documentation. Please note that the only
>> alternative that I observe in practice without design-by-contract is to
>> write contracts in docstrings in *natural language*. Most often, they
>> are just assumed, so the next programmer burns her fingers expecting the
>> contracts to hold when they actually differ from the class or function
>> description, but nobody bothered to update the docstrings (which is a
>> common pitfall in any code base over a longer period of time).
>>
>> *Automatic generation of tests.* Michael wrote:
>>
>>> What might be interesting is somebody wrote a library that does
>>> something more then just adding asserts. For example, one idea might be to
>>> try hooking up a contracts library to hypothesis (or any other library that
>>> does quickcheck-style testing). That might be a good way of partially
>>> addressing the problems up above -- you write out your invariants, and a
>>> testing library extracts that information and uses it to automatically
>>> synthesize interesting test cases.
>>
>>
>> This is the final goal and my main motivation to push for
>> design-by-contract in Python :). There is a whole research community that
>> tries to come up with automatic test generations, and contracts are of
>> great utility there. Mind that generating the tests based on contracts is
>> not trivial: hypothesis just picks elements for each input independently
>> which is a much easier problem. However, preconditions can define how the
>> arguments are *related*. Assume a function takes two numbers as
>> arguments, x and y. If the precondition is y < x < (y + x) * 10, it is not
>> trivial even for this simple example to come up with concrete samples of x
>> and y unless you simply brute-force the problem by densely sampling all the
>> numbers and checking the precondition.
>>
>> I see a chicken-and-egg problem here. If design-by-contract is not widely
>> adopted, there will also be fewer or no libraries for automatic test
>> generation. Honestly, I have absolutely no idea how you could approach
>> automatic generation of test cases without contracts (in one form or the
>> other). For example, how could you automatically mock a class without
>> knowing its invariants?
>>
>> Since generating test cases for functions with non-trivial contracts is
>> hard (and involves collaboration of many people), I don't expect anybody to
>> start even thinking about it if the tool can only be applied to almost
>> anywhere due to lack of contracts. Formal proofs and static analysis are
>> even harder beasts to tame -- and I'd say the argument holds true for them
>> even more.
>>
>> David and Michael, thank you again for your comments! I welcome very much
>> your opinion and any follow-ups as well as from other participants on this
>> mail list.
>>
>> Cheers,
>> Marko
>>
>> On Sat, 15 Sep 2018 at 10:42, Michael Lee <michael.lee.0x2a at gmail.com>
>> wrote:
>>
>>> I just want to point out that you don't need permission from anybody to
>>> start a library. I think developing and popularizing a contracts library is
>>> a reasonable goal -- but that's something you can start doing at any time
>>> without waiting for consensus.
>>>
>>> And if it gets popular enough, maybe it'll be added to the standard
>>> library in some form. That's what happened with attrs, iirc -- it got
>>> fairly popular and demonstrated there was an unfilled niche, and so Python
>>> acquired dataclasses..
>>>
>>>
>>> The contracts make merely tests obsolete that test that the function or
>>>> class actually observes the contracts.
>>>>
>>>
>>> Is this actually the case? Your contracts are only checked when the
>>> function is evaluated, so you'd still need to write that unit test that
>>> confirms the function actually observes the contract. I don't think you
>>> necessarily get to reduce the number of tests you'd need to write.
>>>
>>>
>>> Please let me know what points *do not *convince you that Python needs
>>>> contracts
>>>>
>>>
>>> While I agree that contracts are a useful tool, I don't think they're
>>> going to be necessarily useful for *all* Python programmers. For example,
>>> contracts aren't particularly useful if you're writing fairly
>>> straightforward code with relatively simple invariants.
>>>
>>> I'm also not convinced that libraries where contracts are checked
>>> specifically *at runtime* actually give you that much added power and
>>> impact. For example, you still need to write a decent number of unit tests
>>> to make sure your contracts are being upheld (unless you plan on checking
>>> this by just deploying your code and letting it run, which seems
>>> suboptimal). There's also no guarantee that your contracts will necessarily
>>> be *accurate*. It's entirely possible that your
>>> preconditions/postconditions might hold for every test case you can think
>>> of, but end up failing when running in production due to some edge case
>>> that you missed. (And if you decide to disable those pre/post conditions to
>>> avoid the efficiency hit, you're back to square zero.)
>>>
>>> Or I guess to put it another way -- it seems what all of these contract
>>> libraries are doing is basically adding syntax to try and make adding
>>> asserts in various places more ergonomic, and not much else. I agree those
>>> kinds of libraries can be useful, but I don't think they're necessarily
>>> useful enough to be part of the standard library or to be a technique
>>> Python programmers should automatically use by default.
>>>
>>> What might be interesting is somebody wrote a library that does
>>> something more then just adding asserts. For example, one idea might be to
>>> try hooking up a contracts library to hypothesis (or any other library that
>>> does quickcheck-style testing). That might be a good way of partially
>>> addressing the problems up above -- you write out your invariants, and a
>>> testing library extracts that information and uses it to automatically
>>> synthesize interesting test cases.
>>>
>>> (And of course, what would be very cool is if the contracts could be
>>> verified statically like you can do in languages like dafny -- that way,
>>> you genuinely would be able to avoid writing many kinds of tests and could
>>> have confidence your contracts are upheld. But I understanding implementing
>>> such verifiers are extremely challenging and would probably have too-steep
>>> of a learning curve to be usable by most people anyways.)
>>>
>>> -- Michael
>>>
>>>
>>>
>>> On Fri, Sep 14, 2018 at 11:51 PM, Marko Ristin-Kaufmann <
>>> marko.ristin at gmail.com> wrote:
>>>
>>>> Hi,
>>>> Let me make a couple of practical examples from the work-in-progress (
>>>> https://github.com/Parquery/pypackagery, branch
>>>> mristin/initial-version) to illustrate again the usefulness of the
>>>> contracts and why they are, in my opinion, superior to assertions and unit
>>>> tests.
>>>>
>>>> What follows is a list of function signatures decorated with contracts
>>>> from pypackagery library preceded by a human-readable description of the
>>>> contracts.
>>>>
>>>> The invariants tell us what format to expect from the related string
>>>> properties.
>>>>
>>>> @icontract.inv(lambda self: self.name.strip() == self.name)
>>>> @icontract.inv(lambda self: self.line.endswith("\n"))
>>>> class Requirement:
>>>> """Represent a requirement in requirements.txt."""
>>>>
>>>> def __init__(self, name: str, line: str) -> None:
>>>> """
>>>> Initialize.
>>>>
>>>> :param name: package name
>>>> :param line: line in the requirements.txt file
>>>> """
>>>> ...
>>>>
>>>> The postcondition tells us that the resulting map keys the values on
>>>> their name property.
>>>>
>>>> @icontract.post(lambda result: all(val.name == key for key, val in result.items()))
>>>> def parse_requirements(text: str, filename: str = '<unknown>') -> Mapping[str, Requirement]:
>>>> """
>>>> Parse requirements file and return package name -> package requirement as in requirements.txt
>>>>
>>>> :param text: content of the ``requirements.txt``
>>>> :param filename: where we got the ``requirements.txt`` from (URL or path)
>>>> :return: name of the requirement (*i.e.* pip package) -> parsed requirement
>>>> """
>>>> ...
>>>>
>>>>
>>>> The postcondition ensures that the resulting list contains only unique
>>>> elements. Mind that if you returned a set, the order would have been lost.
>>>>
>>>> @icontract.post(lambda result: len(result) == len(set(result)), enabled=icontract.SLOW)
>>>> def missing_requirements(module_to_requirement: Mapping[str, str],
>>>> requirements: Mapping[str, Requirement]) -> List[str]:
>>>> """
>>>> List requirements from module_to_requirement missing in the ``requirements``.
>>>>
>>>> :param module_to_requirement: parsed ``module_to_requiremnt.tsv``
>>>> :param requirements: parsed ``requirements.txt``
>>>> :return: list of requirement names
>>>> """
>>>> ...
>>>>
>>>> Here is a bit more complex example.
>>>> - The precondition A requires that all the supplied relative paths
>>>> (rel_paths) are indeed relative (as opposed to absolute).
>>>> - The postcondition B ensures that the initial set of paths (given in
>>>> rel_paths) is included in the results.
>>>> - The postcondition C ensures that the requirements in the results are
>>>> the subset of the given requirements.
>>>> - The precondition D requires that there are no missing requirements (*i.e.
>>>> *that each requirement in the given module_to_requirement is also
>>>> defined in the given requirements).
>>>>
>>>> @icontract.pre(lambda rel_paths: all(rel_pth.root == "" for rel_pth in rel_paths)) # A
>>>> @icontract.post(
>>>> lambda rel_paths, result: all(pth in result.rel_paths for pth in rel_paths),
>>>> enabled=icontract.SLOW,
>>>> description="Initial relative paths included") # B
>>>> @icontract.post(
>>>> lambda requirements, result: all(req.name in requirements for req in result.requirements),
>>>> enabled=icontract.SLOW) # C
>>>> @icontract.pre(
>>>> lambda requirements, module_to_requirement: missing_requirements(module_to_requirement, requirements) == [],
>>>> enabled=icontract.SLOW) # D
>>>> def collect_dependency_graph(root_dir: pathlib.Path, rel_paths: List[pathlib.Path],
>>>> requirements: Mapping[str, Requirement],
>>>> module_to_requirement: Mapping[str, str]) -> Package:
>>>>
>>>> """
>>>> Collect the dependency graph of the initial set of python files from the code base.
>>>>
>>>> :param root_dir: root directory of the codebase such as "/home/marko/workspace/pqry/production/src/py"
>>>> :param rel_paths: initial set of python files that we want to package. These paths are relative to root_dir.
>>>> :param requirements: requirements of the whole code base, mapped by package name
>>>> :param module_to_requirement: module to requirement correspondence of the whole code base
>>>> :return: resolved depedendency graph including the given initial relative paths,
>>>> """
>>>>
>>>> I hope these examples convince you (at least a little bit :-)) that
>>>> contracts are easier and clearer to write than asserts. As noted before in
>>>> this thread, you can have the same *behavior* with asserts as long as
>>>> you don't need to inherit the contracts. But the contract decorators make
>>>> it very explicit what conditions should hold *without* having to look
>>>> into the implementation. Moreover, it is very hard to ensure the
>>>> postconditions with asserts as soon as you have a complex control flow since
>>>> you would need to duplicate the assert at every return statement. (You
>>>> could implement a context manager that ensures the postconditions, but a
>>>> context manager is not more readable than decorators and you have to
>>>> duplicate them as documentation in the docstring).
>>>>
>>>> In my view, contracts are also superior to many kinds of tests. As the
>>>> contracts are *always* enforced, they also enforce the correctness
>>>> throughout the program execution whereas the unit tests and doctests only
>>>> cover a list of selected cases. Furthermore, writing the contracts in these
>>>> examples as doctests or unit tests would escape the attention of most less
>>>> experienced programmers which are not used to read unit tests as
>>>> documentation. Finally, these unit tests would be much harder to read than
>>>> the decorators (*e.g.*, the unit test would supply invalid arguments
>>>> and then check for ValueError which is already a much more convoluted piece
>>>> of code than the preconditions and postconditions as decorators. Such
>>>> testing code also lives in a file separate from the original implementation
>>>> making it much harder to locate and maintain).
>>>>
>>>> Mind that the contracts *do not* *replace* the unit tests or the
>>>> doctests. The contracts make merely tests obsolete that test that the
>>>> function or class actually observes the contracts. Design-by-contract helps
>>>> you skip those tests and focus on the more complex ones that test the
>>>> behavior. Another positive effect of the contracts is that they make your
>>>> tests deeper: if you specified the contracts throughout the code base, a
>>>> test of a function that calls other functions in its implementation will
>>>> also make sure that all the contracts of that other functions hold. This
>>>> can be difficult to implement with standard unit test frameworks.
>>>>
>>>> Another aspect of the design-by-contract, which is IMO ignored quite
>>>> often, is the educational one. Contracts force the programmer to actually
>>>> sit down and think *formally* about the inputs and the outputs
>>>> (hopefully?) *before* she starts to implement a function. Since many
>>>> schools use Python to teach programming (especially at high school level),
>>>> I imagine writing contracts of a function to be a very good exercise in
>>>> formal thinking for the students.
>>>>
>>>> Please let me know what points *do not *convince you that Python needs
>>>> contracts (in whatever form -- be it as a standard library, be it as a
>>>> language construct, be it as a widely adopted and collectively maintained
>>>> third-party library). I would be very glad to address these points in my
>>>> next message(s).
>>>>
>>>> Cheers,
>>>> Marko
>>>>
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180922/a682427e/attachment-0001.html>
More information about the Python-ideas
mailing list