From nicoddemus at gmail.com Tue Jun 5 13:52:03 2018 From: nicoddemus at gmail.com (Bruno Oliveira) Date: Tue, 5 Jun 2018 14:52:03 -0300 Subject: [pytest-dev] pytest 3.6.1 Message-ID: pytest 3.6.1 has just been released to PyPI. This is a bug-fix release, being a drop-in replacement. To upgrade:: pip install --upgrade pytest The full changelog is available at http://doc.pytest.org/en/latest/changelog.html. Thanks to all who contributed to this release, among them: * Anthony Sottile * Bruno Oliveira * Jeffrey Rackauckas * Miro Hron?ok * Niklas Meinzer * Oliver Bestwalter * Ronny Pfannschmidt Happy testing, The pytest Development Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From kvas.it at gmail.com Thu Jun 21 05:33:40 2018 From: kvas.it at gmail.com (Vasily Kuznetsov) Date: Thu, 21 Jun 2018 11:33:40 +0200 Subject: [pytest-dev] What is the best practice for tests that install python packages and run their tests? Message-ID: Hello, everyone! Some time ago I wrote a pytest plugin [1] that helps with testing of scripts declared by 'console_scripts' entry point in setup.py. The tests of this plugin install a test package (let's call it TP) and then run TP's tests that are verifying that the scripts declared by TP's setup.py work as expected. TP is installed into the same python environment that runs the outer tests (uninstalled at fixture cleanup) and then the inner tests are run with the same pytest instance that runs the outer tests. This approach is somewhat dirty, but it works fine with disposable virtualenvs that we get from Tox. In the process of trying to package this plugin for Fedora it turned out that my test approach doesn't work with the way they run tests. They use system python so my tests try to install things into system python, don't have permissions and fail (see [2]). I think this could be fixed by creating a virtualenv in tmpdir and using that to install TP and run its tests. However, this new virtualenv would need to have pytest and pytest-console-scripts installed in it in order to run the inner tests. I'm planning to look into the ways to make a virtualenv inherit packages from another virtualenv (or from system python, as would be the case with Fedora package testing) but before I go into all this, I thought I would ask around if maybe there's a better way. Have any of you had situations where your tests needed to install packages and run their tests and how did you go about this? Thanks, Vasily [1]: https://pypi.org/project/pytest-console-scripts/ [2]: https://github.com/kvas-it/pytest-console-scripts/issues/11 -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide.moro at gmail.com Sat Jun 23 01:47:26 2018 From: davide.moro at gmail.com (davide moro) Date: Sat, 23 Jun 2018 07:47:26 +0200 Subject: [pytest-dev] What is the best practice for tests that install python packages and run their tests? In-Reply-To: References: Message-ID: Hi Vasily, I think you could run safely things using docker. With docker no need to install a virtual env (use the simply the system python). You have to provide on the fly an image with preinstalled what you need starting with a Dockerfile and a requirements.txt (with all versions pinned) with a pytest ENTRYPOINT I suppose. You can use a pytest fixture where you build a docker image and later you can docker run it providing the CMD argument in command line. You can also parameterize the fixture and repeating your tests against any operative system (Ubuntu, fedora, alpine, Suse, etc) from any host. Remember the rm option running the image. Alternatively it is fine too a virtual environment in a temp directory but you have to preinstalled what you need, using a requirements.txt file. If there is a bug in your testware be aware because there might be the risk of the outer python environment pollution. Hope it helped. Cheers, davide On Thu, 21 Jun 2018, 11:35 Vasily Kuznetsov, wrote: > Hello, everyone! > > Some time ago I wrote a pytest plugin [1] that helps with testing of > scripts declared by 'console_scripts' entry point in setup.py. The tests of > this plugin install a test package (let's call it TP) and then run TP's > tests that are verifying that the scripts declared by TP's setup.py work as > expected. TP is installed into the same python environment that runs the > outer tests (uninstalled at fixture cleanup) and then the inner tests are > run with the same pytest instance that runs the outer tests. This approach > is somewhat dirty, but it works fine with disposable virtualenvs that we > get from Tox. > > In the process of trying to package this plugin for Fedora it turned out > that my test approach doesn't work with the way they run tests. They use > system python so my tests try to install things into system python, don't > have permissions and fail (see [2]). > > I think this could be fixed by creating a virtualenv in tmpdir and using > that to install TP and run its tests. However, this new virtualenv would > need to have pytest and pytest-console-scripts installed in it in order to > run the inner tests. I'm planning to look into the ways to make a > virtualenv inherit packages from another virtualenv (or from system python, > as would be the case with Fedora package testing) but before I go into all > this, I thought I would ask around if maybe there's a better way. > > Have any of you had situations where your tests needed to install packages > and run their tests and how did you go about this? > > Thanks, > Vasily > > [1]: https://pypi.org/project/pytest-console-scripts/ > [2]: https://github.com/kvas-it/pytest-console-scripts/issues/11 > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bubenkoff at gmail.com Sat Jun 23 10:51:27 2018 From: bubenkoff at gmail.com (Anatoly Bubenkov) Date: Sat, 23 Jun 2018 15:51:27 +0100 Subject: [pytest-dev] What is the best practice for tests that install python packages and run their tests? In-Reply-To: References: Message-ID: Hey Vasily, I think pytest-cloud approach can be useful for you, as it spreads virtual env to all test nodes automatically On Thu, Jun 21, 2018 at 11:35 AM Vasily Kuznetsov wrote: > Hello, everyone! > > Some time ago I wrote a pytest plugin [1] that helps with testing of > scripts declared by 'console_scripts' entry point in setup.py. The tests of > this plugin install a test package (let's call it TP) and then run TP's > tests that are verifying that the scripts declared by TP's setup.py work as > expected. TP is installed into the same python environment that runs the > outer tests (uninstalled at fixture cleanup) and then the inner tests are > run with the same pytest instance that runs the outer tests. This approach > is somewhat dirty, but it works fine with disposable virtualenvs that we > get from Tox. > > In the process of trying to package this plugin for Fedora it turned out > that my test approach doesn't work with the way they run tests. They use > system python so my tests try to install things into system python, don't > have permissions and fail (see [2]). > > I think this could be fixed by creating a virtualenv in tmpdir and using > that to install TP and run its tests. However, this new virtualenv would > need to have pytest and pytest-console-scripts installed in it in order to > run the inner tests. I'm planning to look into the ways to make a > virtualenv inherit packages from another virtualenv (or from system python, > as would be the case with Fedora package testing) but before I go into all > this, I thought I would ask around if maybe there's a better way. > > Have any of you had situations where your tests needed to install packages > and run their tests and how did you go about this? > > Thanks, > Vasily > > [1]: https://pypi.org/project/pytest-console-scripts/ > [2]: https://github.com/kvas-it/pytest-console-scripts/issues/11 > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From obeythetestinggoat at gmail.com Sun Jun 24 15:17:45 2018 From: obeythetestinggoat at gmail.com (Harry Percival) Date: Sun, 24 Jun 2018 20:17:45 +0100 Subject: [pytest-dev] tips on writing a plugin for a third-party test framework? Message-ID: hi there, at work we use a test framework called "Contexts": http://contexts.readthedocs.io/en/v0.11.2/ it's basically a syntax for declaring tests using a given/when/then kind of syntax, using classes and special methods. as a way of structuring your tests it's... fine, once you get used to it. as a test runner and assertion framework it's fairly primitive. so i'd like to write a pytest plugin for it that basically just re-uses the parts of it that know how to recognise given/when/then bits, and then just use pytest as the test runner and assertion library. any pointers for how to get started? i thought i'd get a prototype started just based on `conftest.py`, and I've found the `pytest_pycollect_makeitem` which seems to be a good place to intercept for collecting tests, but I don't know what it's supposed to return (an "Item"? what's that? docs are minimal) or how to provide info about setup and teardown methods, which I guess would need to become fixtures somehow, but how would I associate them with a test method? maybe there's some example code somewhere i could look at? i tried peeking at pytest-bdd but it seems to be doing things at a lower level... hp -------------- next part -------------- An HTML attachment was scrubbed... URL: From obeythetestinggoat at gmail.com Thu Jun 28 05:05:26 2018 From: obeythetestinggoat at gmail.com (Harry Percival) Date: Thu, 28 Jun 2018 10:05:26 +0100 Subject: [pytest-dev] tips on writing a plugin for a third-party test framework? In-Reply-To: References: Message-ID: PS a couple of people have already pointed me towards some useful example repos: - Sybil: https://github.com/cjw296/sybil/blob/master/sybil/integration/pytest.py - Pytest-CPP: https://github.com/pytest-dev/pytest-cpp/blob/master/pytest_cpp/plugin.py <-- really nice and simple :) so I've been able to make a start, and i actually already have a hacky first cut. all good! On Sun, Jun 24, 2018 at 8:17 PM Harry Percival wrote: > hi there, > > at work we use a test framework called "Contexts": > http://contexts.readthedocs.io/en/v0.11.2/ > > it's basically a syntax for declaring tests using a given/when/then kind > of syntax, using classes and special methods. > > as a way of structuring your tests it's... fine, once you get used to it. > as a test runner and assertion framework it's fairly primitive. so i'd > like to write a pytest plugin for it that basically just re-uses the parts > of it that know how to recognise given/when/then bits, and then just use > pytest as the test runner and assertion library. > > any pointers for how to get started? i thought i'd get a prototype started > just based on `conftest.py`, and I've found the `pytest_pycollect_makeitem` > which seems to be a good place to intercept for collecting tests, but I > don't know what it's supposed to return (an "Item"? what's that? docs are > minimal) or how to provide info about setup and teardown methods, which I > guess would need to become fixtures somehow, but how would I associate them > with a test method? > > maybe there's some example code somewhere i could look at? i tried > peeking at pytest-bdd but it seems to be doing things at a lower level... > > hp > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ringo.de.smet at ontoforce.com Thu Jun 28 08:50:39 2018 From: ringo.de.smet at ontoforce.com (Ringo De Smet) Date: Thu, 28 Jun 2018 14:50:39 +0200 Subject: [pytest-dev] tips on writing a plugin for a third-party test framework? In-Reply-To: References: Message-ID: Harry, I also implemented a custom pytest plugin which I can't open source at this moment. But what I did find is that the traceback filtering in case of test failures doesn't work correctly when returning custom test `Item` objects. See this mail thread from the archives about it: https://mail.python.org/pipermail/pytest-dev/2018-March/004372.html Hope it helps you too. Ringo On 28 June 2018 at 11:05, Harry Percival wrote: > PS a couple of people have already pointed me towards some useful example > repos: > > - Sybil: https://github.com/cjw296/sybil/blob/master/sybil/ > integration/pytest.py > - Pytest-CPP: https://github.com/pytest-dev/ > pytest-cpp/blob/master/pytest_cpp/plugin.py <-- really nice and simple :) > > so I've been able to make a start, and i actually already have a hacky > first cut. all good! > > > On Sun, Jun 24, 2018 at 8:17 PM Harry Percival < > obeythetestinggoat at gmail.com> wrote: > >> hi there, >> >> at work we use a test framework called "Contexts": >> http://contexts.readthedocs.io/en/v0.11.2/ >> >> it's basically a syntax for declaring tests using a given/when/then kind >> of syntax, using classes and special methods. >> >> as a way of structuring your tests it's... fine, once you get used to >> it. as a test runner and assertion framework it's fairly primitive. so >> i'd like to write a pytest plugin for it that basically just re-uses the >> parts of it that know how to recognise given/when/then bits, and then just >> use pytest as the test runner and assertion library. >> >> any pointers for how to get started? i thought i'd get a prototype >> started just based on `conftest.py`, and I've found the >> `pytest_pycollect_makeitem` which seems to be a good place to intercept for >> collecting tests, but I don't know what it's supposed to return (an >> "Item"? what's that? docs are minimal) or how to provide info about >> setup and teardown methods, which I guess would need to become fixtures >> somehow, but how would I associate them with a test method? >> >> maybe there's some example code somewhere i could look at? i tried >> peeking at pytest-bdd but it seems to be doing things at a lower level... >> >> hp >> >> >> > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > > -- *Ringo De Smet* ringo.de.smet at ontoforce.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From obeythetestinggoat at gmail.com Fri Jun 29 12:34:18 2018 From: obeythetestinggoat at gmail.com (Harry Percival) Date: Fri, 29 Jun 2018 17:34:18 +0100 Subject: [pytest-dev] tips on writing a plugin for a third-party test framework? In-Reply-To: References: Message-ID: got a first cut up and running everyone! thanks for the help, here and on twitter and elsewhere :) https://pypi.org/project/pytest-contexts/ On Thu, Jun 28, 2018 at 1:50 PM Ringo De Smet wrote: > Harry, > > I also implemented a custom pytest plugin which I can't open source at > this moment. But what I did find is that the traceback filtering in case of > test failures doesn't work correctly when returning custom test `Item` > objects. See this mail thread from the archives about it: > > https://mail.python.org/pipermail/pytest-dev/2018-March/004372.html > > Hope it helps you too. > > Ringo > > > On 28 June 2018 at 11:05, Harry Percival > wrote: > >> PS a couple of people have already pointed me towards some useful example >> repos: >> >> - Sybil: >> https://github.com/cjw296/sybil/blob/master/sybil/integration/pytest.py >> - Pytest-CPP: >> https://github.com/pytest-dev/pytest-cpp/blob/master/pytest_cpp/plugin.py >> <-- really nice and simple :) >> >> so I've been able to make a start, and i actually already have a hacky >> first cut. all good! >> >> >> On Sun, Jun 24, 2018 at 8:17 PM Harry Percival < >> obeythetestinggoat at gmail.com> wrote: >> >>> hi there, >>> >>> at work we use a test framework called "Contexts": >>> http://contexts.readthedocs.io/en/v0.11.2/ >>> >>> it's basically a syntax for declaring tests using a given/when/then kind >>> of syntax, using classes and special methods. >>> >>> as a way of structuring your tests it's... fine, once you get used to >>> it. as a test runner and assertion framework it's fairly primitive. so >>> i'd like to write a pytest plugin for it that basically just re-uses the >>> parts of it that know how to recognise given/when/then bits, and then just >>> use pytest as the test runner and assertion library. >>> >>> any pointers for how to get started? i thought i'd get a prototype >>> started just based on `conftest.py`, and I've found the >>> `pytest_pycollect_makeitem` which seems to be a good place to intercept for >>> collecting tests, but I don't know what it's supposed to return (an >>> "Item"? what's that? docs are minimal) or how to provide info about >>> setup and teardown methods, which I guess would need to become fixtures >>> somehow, but how would I associate them with a test method? >>> >>> maybe there's some example code somewhere i could look at? i tried >>> peeking at pytest-bdd but it seems to be doing things at a lower level... >>> >>> hp >>> >>> >>> >> _______________________________________________ >> pytest-dev mailing list >> pytest-dev at python.org >> https://mail.python.org/mailman/listinfo/pytest-dev >> >> > > > -- > *Ringo De Smet* > ringo.de.smet at ontoforce.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: