From jacek.generowicz at cern.ch Thu Dec 8 08:30:02 2011 From: jacek.generowicz at cern.ch (Jacek Generowicz) Date: Thu, 8 Dec 2011 08:30:02 +0100 Subject: [py-dev] QuickCheck plugin Message-ID: <31C08DE0-47F0-4662-8B8E-B5CC89CE8687@cern.ch> Hi, QuickCheck is a property-based testing framework originating in Haskell. Its 3 key concepts are: 1. Tests are expressed as properties: invariants which should be satisfied for any appropriate test data. 2. The test data are randomly generated and fed to the property. 3. (this is the really interesting stroke of genius) When some data are found for which the test fails, QuickCheck attempts to simplify the data to obtain the minimal example that still fails. My questions are: 1. Are you aware of any work being done on integrating the QuickCheck idea with pytest? 2. QuickCheck generates new test invocations in response to failing tests; Pytest collects tests *before* they are run. Would these two seemingly conflicting features prose problems in trying to implement a QuickCheck plugin for pytest? 3. Would you have any general advice on how one might approach providing a QuickCheck clone on top of pytest? Thanks. From Ronny.Pfannschmidt at gmx.de Thu Dec 8 11:15:06 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 08 Dec 2011 11:15:06 +0100 Subject: [py-dev] QuickCheck plugin In-Reply-To: <31C08DE0-47F0-4662-8B8E-B5CC89CE8687@cern.ch> References: <31C08DE0-47F0-4662-8B8E-B5CC89CE8687@cern.ch> Message-ID: <4EE08E2A.7000806@gmx.de> On 12/08/2011 08:30 AM, Jacek Generowicz wrote: > Hi, > > > QuickCheck is a property-based testing framework originating in > Haskell. Its 3 key concepts are: > > 1. Tests are expressed as properties: invariants which should be > satisfied for any appropriate test data. > > 2. The test data are randomly generated and fed to the property. > > 3. (this is the really interesting stroke of genius) When some data > are found for which the test fails, QuickCheck attempts to simplify > the data to obtain the minimal example that still fails. > > > My questions are: > > 1. Are you aware of any work being done on integrating the QuickCheck > idea with pytest? so far there is no direct work being done the generate_tests hooks and parametrize are going into something that could be a good base for approaching randomized test generators > > 2. QuickCheck generates new test invocations in response to failing > tests; Pytest collects tests *before* they are run. Would these two > seemingly conflicting features prose problems in trying to implement a > QuickCheck plugin for pytest? pytest itself has a strict collect first, run then model however i think it might make sense to extend the looponfail mode of the xdist plugin the basic idea is that the next batch fo generated items would be executed in the next looponfail iteration i don't know how well this can sit with the design of xdist/looponfail, but it seems doable to have it collect new items for generative functions > > 3. Would you have any general advice on how one might approach > providing a QuickCheck clone on top of pytest? > * start with a library that can simply generate those randomized values -> instantly usable with parametrize+generate_tests * continue extending it with creating simplifications * extend reports with extra data about * teach the looponfail runner to generate different items based on failure (you btw, it would be nice if there was some basic way to recreate a test with the same conditions based on some kind of id/identification best regards, Ronny Pfannschmidt > > Thanks. > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From sienkiew at stsci.edu Fri Dec 9 16:43:22 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Fri, 09 Dec 2011 10:43:22 -0500 Subject: [py-dev] QuickCheck plugin In-Reply-To: <31C08DE0-47F0-4662-8B8E-B5CC89CE8687@cern.ch> References: <31C08DE0-47F0-4662-8B8E-B5CC89CE8687@cern.ch> Message-ID: <4EE22C9A.3040305@stsci.edu> On 12/08/11 02:30, Jacek Generowicz wrote: > > 3. Would you have any general advice on how one might approach > providing a QuickCheck clone on top of pytest? From your description of QuickCheck, I suggest a modular approach where you do not implement QuickCheck inside py.test, but instead use py.test as a tool that performs only part of the work. Consider this approach: - QuickCheck reads the definition of the properties. - It generates the random test data. - It writes a python file that contains tests suitable for py.test to run. - It invokes py.test to actually run the tests. Use a py.test plugin to report test statuses in a form that QuickCheck can easily read. (I can show you an example if you need help with this.) - QuickCheck reads the report file to find out how the tests worked. It then iterates as you described in your original email. As you described it, QuickCheck has a fundamentally different view of tests than py.test does. Instead of trying to force the QuickCheck model into the py.test structure, this alternative uses a control program to implement the main structure of QuickCheck. That control program uses py.test as a subroutine to perform the part that py.test is really good at. You might import pytest and call functions in it, or you might do something as simple as os.system("py.test tmp.py") Mark S. From holger at merlinux.eu Fri Dec 16 12:56:53 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 16 Dec 2011 11:56:53 +0000 Subject: [py-dev] pytest-2.2.1: bug fixes, better teardowns Message-ID: <20111216115653.GF27920@merlinux.eu> pytest-2.2.1: bug fixes, better teardowns =========================================================================== pytest-2.2.1 is a minor backward-compatible release of the the py.test testing tool. It contains bug fixes and little improvements, including documentation fixes. If you are using the distributed testing plugin make sure to upgrade it to pytest-xdist-1.8. For general information see here: http://pytest.org/ To install or upgrade pytest: pip install -U pytest # or easy_install -U pytest Special thanks for helping on this release to Ronny Pfannschmidt, Jurko Gospodnetic and Ralf Schmitt. best, holger krekel Changes between 2.2.0 and 2.2.1 ---------------------------------------- - fix issue99 (in pytest and py) internallerrors with resultlog now produce better output - fixed by normalizing pytest_internalerror input arguments. - fix issue97 / traceback issues (in pytest and py) improve traceback output in conjunction with jinja2 and cython which hack tracebacks - fix issue93 (in pytest and pytest-xdist) avoid "delayed teardowns": the final test in a test node will now run its teardown directly instead of waiting for the end of the session. Thanks Dave Hunt for the good reporting and feedback. The pytest_runtest_protocol as well as the pytest_runtest_teardown hooks now have "nextitem" available which will be None indicating the end of the test run. - fix collection crash due to unknown-source collected items, thanks to Ralf Schmitt (fixed by depending on a more recent pylib) From holger at merlinux.eu Wed Dec 21 09:41:10 2011 From: holger at merlinux.eu (holger krekel) Date: Wed, 21 Dec 2011 08:41:10 +0000 Subject: [py-dev] tox-1.3: virtualenv-based test run automizer Message-ID: <20111221084110.GO27920@merlinux.eu> tox 1.3: the virtualenv-based test run automatizer =========================================================================== I am happy to announce tox 1.3, containing a few improvements over 1.2. TOX automates tedious test activities driven from a simple ``tox.ini`` file, including: * creation and management of different virtualenv environments with different Python interpreters * packaging and installing your package into each of them * running your test tool of choice, be it nose, py.test or unittest2 or other tools such as "sphinx" doc checks * testing dev packages against each other without needing to upload to PyPI Docs and examples are at: http://tox.testrun.org/ Installation: pip install -U tox code hosting and issue tracking on bitbucket: http://bitbucket.org/hpk42/tox best, Holger Krekel 1.3 ----------------- - fix: allow to specify wildcard filesystem paths when specifiying dependencies such that tox searches for the highest version - fix issue21: clear PIP_REQUIRES_VIRTUALENV which avoids pip installing to the wrong environment, thanks to bb's streeter - make the install step honour a testenv's setenv setting (thanks Ralf Schmitt)