From ringo.de.smet at ontoforce.com Wed Apr 4 07:22:54 2018 From: ringo.de.smet at ontoforce.com (Ringo De Smet) Date: Wed, 4 Apr 2018 13:22:54 +0200 Subject: [pytest-dev] Printing plugin help and skip tests Message-ID: Hello, I'm working on a custom (private) plugin vary tailored to our product. The plugin adapts its feature set based on the abilities of the internal server it connects to. I was thinking of implementing an option in my plugin, e.g. `--show-our-superduper-help`. This could go out to the server, fetch it's config and print the abilities which can be used in the test suite for that server. I looked at how the regular `--help` is implemented: https://github.com/pytest-dev/pytest/blob/71a7b3c062d51c61ee8b1ac479b1260d541918c6/_pytest/helpconfig.py#L46-L47 It uses the HelpAction which raises a PrintHelp exception to shortcut on running the test suite. Not really fond of implementing something similar. How could I get a similar behaviour for only printing the (server specific) config of my plugin? Thanks, *Ringo De Smet* ringo.de.smet at ontoforce.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Thu Apr 5 14:49:46 2018 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 05 Apr 2018 20:49:46 +0200 Subject: [pytest-dev] Printing plugin help and skip tests In-Reply-To: References: Message-ID: Hi Ringo, On Wed, Apr 04 2018, Ringo De Smet wrote: > I was thinking of implementing an option in my plugin, e.g. > `--show-our-superduper-help`. > This could go out to the server, fetch it's config and print the abilities > which can be used in the test suite for that server. > > I looked at how the regular `--help` is implemented: > > https://github.com/pytest-dev/pytest/blob/71a7b3c062d51c61ee8b1ac479b1260d541918c6/_pytest/helpconfig.py#L46-L47 > > It uses the HelpAction which raises a PrintHelp exception to shortcut on > running the test suite. > > Not really fond of implementing something similar. How could I get a > similar behaviour for only printing the (server specific) config of my > plugin? You could look at how --fixtures is implemented, that's probably what I would follow if I wanted to do something similar. Search for "showfixtures" in _pytest/python.py. Basically add the option like normal in pytest_addoption() and then implement pytest_cmdline_main() to look for the option and handle it. If the option is not there just return None from it and the normal pytest will continue to run. Hope that helps, Floris From opensource at ronnypfannschmidt.de Sun Apr 8 06:28:17 2018 From: opensource at ronnypfannschmidt.de (RonnyPfannschmidt) Date: Sun, 8 Apr 2018 12:28:17 +0200 Subject: [pytest-dev] preparing a set of breaking changes Message-ID: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> Hi everyone, i wold like to get started with a long needed spring cleaning of things that are unfixable stange/broken since dozens of years * removal of yield tests, ever since collection and test running are no longer connected, their setupstate has been fundamentally broken for anything but test so simple, one should use parametrize * removal of the pytest namespace hook and setting up the pytest config in a way that is actially correct on xdist (currently pytest-xdist is unable to create config objects on workers in a consistent manner, and we leak data in strange ways * removal of the dozns and dozens of compat properties/hackish aliases across the codebase ? -> getting rid of customclasses on collectors * after the mark merge: introduce an actual FunctionDefinition node that's visible in the collection tree and ensure metafunc uses that as base for parameterization, * detangle node construction - the amount of spaghetti, in particular wrt sessions, fs collectors & co has an unbearable amount of stones in it for unit and functional testing -- Ronny From flub at devork.be Sun Apr 8 08:40:14 2018 From: flub at devork.be (Floris Bruynooghe) Date: Sun, 08 Apr 2018 14:40:14 +0200 Subject: [pytest-dev] preparing a set of breaking changes In-Reply-To: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> References: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> Message-ID: <87sh8527o1.fsf@powell.devork.be> Hi Ronny, On Sun 08 Apr 2018 at 12:28 +0200, RonnyPfannschmidt wrote: > i wold like to get started with a long needed spring cleaning of things > that are unfixable stange/broken since dozens of years > > * removal of yield tests, ever since collection and test running are no > longer connected, their setupstate has been fundamentally broken for > anything but test so simple, one should use parametrize Sounds good! Do we already have deprecation warnings for this? If not should we add them ASAP? > * removal of the pytest namespace hook and setting up the pytest config > in a way that is actially correct on xdist (currently pytest-xdist is > unable to create config objects on workers in a consistent manner, > and we leak data in strange ways I guess pytest_namespace was nice for internal plugins but does not really scale to many external plugins. I used to be against this but maybe you're right. Anyway, need to start deprecating this right away if we want to do this! > * removal of the dozns and dozens of compat properties/hackish aliases > across the codebase Sounds great, modulo the deprecation warnings thing. > ? -> getting rid of customclasses on collectors > * after the mark merge: introduce an actual FunctionDefinition node > that's visible in the collection tree and ensure metafunc uses that as > base for parameterization, > * detangle node construction - the amount of spaghetti, in particular > wrt sessions, fs collectors & co has an unbearable amount of stones in > it for unit and functional testing This all sounds worthwhile. Cheers, Floris From flub at devork.be Sun Apr 8 08:40:17 2018 From: flub at devork.be (Floris Bruynooghe) Date: Sun, 08 Apr 2018 14:40:17 +0200 Subject: [pytest-dev] preparing a set of breaking changes In-Reply-To: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> References: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> Message-ID: <87r2np27ny.fsf@powell.devork.be> Hi Ronny, On Sun 08 Apr 2018 at 12:28 +0200, RonnyPfannschmidt wrote: > i wold like to get started with a long needed spring cleaning of things > that are unfixable stange/broken since dozens of years > > * removal of yield tests, ever since collection and test running are no > longer connected, their setupstate has been fundamentally broken for > anything but test so simple, one should use parametrize Sounds good! Do we already have deprecation warnings for this? If not should we add them ASAP? > * removal of the pytest namespace hook and setting up the pytest config > in a way that is actially correct on xdist (currently pytest-xdist is > unable to create config objects on workers in a consistent manner, > and we leak data in strange ways I guess pytest_namespace was nice for internal plugins but does not really scale to many external plugins. I used to be against this but maybe you're right. Anyway, need to start deprecating this right away if we want to do this! > * removal of the dozns and dozens of compat properties/hackish aliases > across the codebase Sounds great, modulo the deprecation warnings thing. > ? -> getting rid of customclasses on collectors > * after the mark merge: introduce an actual FunctionDefinition node > that's visible in the collection tree and ensure metafunc uses that as > base for parameterization, > * detangle node construction - the amount of spaghetti, in particular > wrt sessions, fs collectors & co has an unbearable amount of stones in > it for unit and functional testing This all sounds worthwhile. Cheers, Floris From dan.nealschneider at gmail.com Wed Apr 11 00:57:27 2018 From: dan.nealschneider at gmail.com (Dan Nealschneider) Date: Tue, 10 Apr 2018 21:57:27 -0700 Subject: [pytest-dev] preparing a set of breaking changes In-Reply-To: <87r2np27ny.fsf@powell.devork.be> References: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> <87r2np27ny.fsf@powell.devork.be> Message-ID: Ohhh removal of yield tests is great. I think I have a memory-use reduction commit that broke for yield tests :) On Sun, Apr 8, 2018 at 5:40 AM, Floris Bruynooghe wrote: > Hi Ronny, > > On Sun 08 Apr 2018 at 12:28 +0200, RonnyPfannschmidt wrote: > > i wold like to get started with a long needed spring cleaning of things > > that are unfixable stange/broken since dozens of years > > > > * removal of yield tests, ever since collection and test running are no > > longer connected, their setupstate has been fundamentally broken for > > anything but test so simple, one should use parametrize > > Sounds good! Do we already have deprecation warnings for this? If not > should we add them ASAP? > > > * removal of the pytest namespace hook and setting up the pytest config > > in a way that is actially correct on xdist (currently pytest-xdist is > > unable to create config objects on workers in a consistent manner, > > and we leak data in strange ways > > I guess pytest_namespace was nice for internal plugins but does not > really scale to many external plugins. I used to be against this but > maybe you're right. Anyway, need to start deprecating this right away > if we want to do this! > > > * removal of the dozns and dozens of compat properties/hackish aliases > > across the codebase > > Sounds great, modulo the deprecation warnings thing. > > > -> getting rid of customclasses on collectors > > * after the mark merge: introduce an actual FunctionDefinition node > > that's visible in the collection tree and ensure metafunc uses that as > > base for parameterization, > > * detangle node construction - the amount of spaghetti, in particular > > wrt sessions, fs collectors & co has an unbearable amount of stones in > > it for unit and functional testing > > This all sounds worthwhile. > > > Cheers, > Floris > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdent at anticdent.org Wed Apr 11 06:46:40 2018 From: cdent at anticdent.org (Chris Dent) Date: Wed, 11 Apr 2018 11:46:40 +0100 (BST) Subject: [pytest-dev] preparing a set of breaking changes In-Reply-To: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> References: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> Message-ID: On Sun, 8 Apr 2018, RonnyPfannschmidt wrote: > * removal of yield tests, ever since collection and test running are no > longer connected, their setupstate has been fundamentally broken for > anything but test so simple, one should use parametrize Not saying it's wrong to see them go, but back in perhaps 2010 (not really sure of the exact time) pytest's yield tests are what made me fall in love with pytest. That was back when pytest was still a thing that did some collecting of stuff named 'test_*' that has 'assert' calls in it, and not much else. I loved that. All the fixtures, decorators, etc that we have now are nicely powerful and useful, but back when you couldn't do much in tests it meant that during TDD you couldn't do much in code either, and that was a great thing. /me raises a glass to yield tests, bon voyage -- Chris Dent ????? https://anticdent.org/ freenode: cdent tw: @anticdent From rpfannsc at redhat.com Wed Apr 11 08:07:25 2018 From: rpfannsc at redhat.com (Ronny Pfannschmidt) Date: Wed, 11 Apr 2018 14:07:25 +0200 Subject: [pytest-dev] preparing a set of breaking changes In-Reply-To: References: <490b80a3-f050-c03b-cac9-140dd861b2ed@ronnypfannschmidt.de> Message-ID: in 2010 yield tests where likely still working correctly they where fundamentally broken ever since pytest introduced the collection phase -- Ronny 2018-04-11 12:46 GMT+02:00 Chris Dent : > On Sun, 8 Apr 2018, RonnyPfannschmidt wrote: > > * removal of yield tests, ever since collection and test running are no >> longer connected, their setupstate has been fundamentally broken for >> anything but test so simple, one should use parametrize >> > > Not saying it's wrong to see them go, but back in perhaps 2010 (not > really sure of the exact time) pytest's yield tests are what made me > fall in love with pytest. > > That was back when pytest was still a thing that did some collecting > of stuff named 'test_*' that has 'assert' calls in it, and not much > else. > > I loved that. All the fixtures, decorators, etc that we have now are > nicely powerful and useful, but back when you couldn't do much in > tests it meant that during TDD you couldn't do much in code either, > and that was a great thing. > > /me raises a glass to yield tests, bon voyage > > -- > Chris Dent ????? https://anticdent.org/ > freenode: cdent tw: @anticdent > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide.moro at gmail.com Thu Apr 12 03:45:41 2018 From: davide.moro at gmail.com (davide moro) Date: Thu, 12 Apr 2018 09:45:41 +0200 Subject: [pytest-dev] new pytest-dev plugin proposal Message-ID: Hi, I asked permission to my company for transferring the ownership of github repositories that might be interesting for the testing community to the pytest-dev organization and they said yes. Talking before with Bruno we thought that the following one might be a candidate: * https://github.com/tierratelematics/pytest-play There is a a cookiecutter scaffolding tool based on pytest providing a working selenium setup, examples with pytest-bdd (it is not a pytest plugin): * https://github.com/tierratelematics/cookiecutter-qa Let me know which of them could be added to the pytest-dev organization. Thanks. Bests, davide -- Linkedin: https://it.linkedin.com/in/davidemoro82 Twitter: https://twitter.com/davidemoro Blog: http://davidemoro.blogspot.it/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicoddemus at gmail.com Thu Apr 12 06:57:25 2018 From: nicoddemus at gmail.com (Bruno Oliveira) Date: Thu, 12 Apr 2018 10:57:25 +0000 Subject: [pytest-dev] new pytest-dev plugin proposal In-Reply-To: References: Message-ID: Hi Davide, +1 to both. Now we need another +1 from some other core dev to start the transferring process. :) Cheers, Bruno. On Thu, Apr 12, 2018 at 4:46 AM davide moro wrote: > Hi, > > I asked permission to my company for transferring the ownership of github > repositories that might be interesting for the testing community to the > pytest-dev organization and they said yes. > > Talking before with Bruno we thought that the following one might be a > candidate: > * https://github.com/tierratelematics/pytest-play > > There is a a cookiecutter scaffolding tool based on pytest providing a > working selenium setup, examples with pytest-bdd (it is not a pytest > plugin): > * https://github.com/tierratelematics/cookiecutter-qa > > Let me know which of them could be added to the pytest-dev organization. > > Thanks. > > Bests, > > davide > > > -- > Linkedin: https://it.linkedin.com/in/davidemoro82 > Twitter: https://twitter.com/davidemoro > Blog: http://davidemoro.blogspot.it/ > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at the-compiler.org Mon Apr 16 01:50:12 2018 From: me at the-compiler.org (Florian Bruhin) Date: Mon, 16 Apr 2018 07:50:12 +0200 Subject: [pytest-dev] Unsubscribing from pytest-dev/pytest notifications Message-ID: <20180416055012.qufdcz3fiuzhy3cv@hooch.localdomain> Hi, (I accidentally sent this to pytest-dev at pytest.org rather than @python.org originally. It didn't bounce, but I have no idea where it went :D) this is a difficult step to take, but during three sleepless nights because of health issues last week (don't worry, it's getting better now) I realized a lot of things (also see this thread on Twitter: [1]) [1] https://twitter.com/the_compiler/status/984356385214394368 I really wish GitHub had a way to subscribe to issue labels, but until that happens, I need to unsubscribe from pytest-dev/pytest (which is what I just did) - I'm spending too much time working on things in pytest I'm somewhat interested in, but not really passionate about. This is probably best explained with some back story: I got into pytest because I started using it with qutebrowser[2] (which is still my main OSS project), started contributing on plugins (mostly pytest-qt) because I was missing some features, and got into the pytest community. Somehow I ended up first getting regarded as a contributor, later as a core contributor (or maintainer) - while I still feel like I have no idea of pytest's code base :D [2] https://www.qutebrowser.org/ There's a list of things I'm still very much interested in - if something happens in those areas, please don't hesitate to contact/mention/request a review from me, I'd very much like to hear about those: - Giving trainings and talks (probably what I liked doing most so far!) - Anything related to user-facing changes: The API, output, etc. I am still an avid user of pytest, and thanks to trainings and other involvement feel like I have a picture of what other users want, too. - The community, which is awesome! I'll still stay on this mailinglist and the IRC channel, and I'd very much like to go to another development sprint in the future. - Testing (pre-)releases - Some plugins, especially pytest-qt - Any code I've written Things I'm explicitly not interested in (and where I never really did anything anyways): - Anything internal and complex in the codebase: Markers, fixtures, pytest.code, capturing, etc. etc. - you get the idea. I'd like to keep my maintainer/commit/whatever flag, and I might revisit this in 1-2 years, where I might have more time on my hand for open-source work. If there's consensus that I should drop it, I'm not going to say no either, though. (If you want to answer something along the lines of "get well soon" - those replies are very much appreciated, but please consider sending them to me directly instead of the mailinglist, to cut down "noise" for others) Florian -- https://www.qutebrowser.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc I love long mails! | https://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From flub at devork.be Tue Apr 17 16:22:04 2018 From: flub at devork.be (Floris Bruynooghe) Date: Tue, 17 Apr 2018 22:22:04 +0200 Subject: [pytest-dev] new pytest-dev plugin proposal In-Reply-To: References: Message-ID: <87zi21ziqb.fsf@powell.devork.be> Hello, On Thu 12 Apr 2018 at 10:57 +0000, Bruno Oliveira wrote: > Hi Davide, > > +1 to both. > > Now we need another +1 from some other core dev to start the transferring > process. :) > > Cheers, > Bruno. > > On Thu, Apr 12, 2018 at 4:46 AM davide moro wrote: > >> Hi, >> >> I asked permission to my company for transferring the ownership of github >> repositories that might be interesting for the testing community to the >> pytest-dev organization and they said yes. Cool! >> Talking before with Bruno we thought that the following one might be a >> candidate: >> * https://github.com/tierratelematics/pytest-play +1 on this >> There is a a cookiecutter scaffolding tool based on pytest providing a >> working selenium setup, examples with pytest-bdd (it is not a pytest >> plugin): >> * https://github.com/tierratelematics/cookiecutter-qa I'm less sure about this one, it looks like pytest is not really the core thing in this project. It's also not really a pytest plugin, but yes I know about cookiecutter-pytest-plugin. But y'all think it fit's just fine that ok with me. Cheers, Floris From flub at devork.be Tue Apr 17 16:33:42 2018 From: flub at devork.be (Floris Bruynooghe) Date: Tue, 17 Apr 2018 22:33:42 +0200 Subject: [pytest-dev] Adding plugins to pytest-dev Message-ID: <87wox5zi6x.fsf@powell.devork.be> Hi all, While just looking at https://github.com/pytest-dev/pytest/blob/master/CONTRIBUTING.rst#submitting-plugins-to-pytest-dev I had some observations: 1. It says one of the objectives of having plugins under pytest-dev is "Sharing some of the maintenance responsibility (in case a maintainer no longer wishes to maintain a plugin)". I found this somewhat surprisingly worded. I thought it was more "protect the pytest community in case a maintainer goes AWOL". I think that's a significant difference, maintainers aren't supposed to think they can move the plugin there if they no longer feel like actively maintaining it. 2. The other thing this says is "We recommend that each plugin has at least three people who have the right to release to PyPI." I think that's somewhat light and does not ensure a package can be released if a maintainer does go AWOL. Could we change that to something like "each plugin must have at least three people from https://github.com/orgs/pytest-dev/teams/core/members who have the right to release to PyPI" [0]? Practically we should probably include a list of pypi usernames to that paragraph since there's not mapping otherwise. Any opinions on these? Cheers, Floris [0] Bonus points for whoever contributes a teams functionality to warehouse! From flub at devork.be Tue Apr 17 16:55:48 2018 From: flub at devork.be (Floris Bruynooghe) Date: Tue, 17 Apr 2018 22:55:48 +0200 Subject: [pytest-dev] Unsubscribing from pytest-dev/pytest notifications In-Reply-To: <20180416055012.qufdcz3fiuzhy3cv@hooch.localdomain> References: <20180416055012.qufdcz3fiuzhy3cv@hooch.localdomain> Message-ID: <87tvs9zh63.fsf@powell.devork.be> Hey Florian, On Mon 16 Apr 2018 at 07:50 +0200, Florian Bruhin wrote: > this is a difficult step to take, but during three sleepless nights > because of health issues last week (don't worry, it's getting better > now) I realized a lot of things (also see this thread on Twitter: [1]) Thanks for being so open about this. > I really wish GitHub had a way to subscribe to issue labels, but until > that happens, I need to unsubscribe from pytest-dev/pytest (which is > what I just did) That's no problem at all. I've not been receiving email for github activity in ages, and that blue dot on github is basically a permanent feature. Volume is pretty high (to me), occasionally I try to look at it but I mostly also rely on those of you who do keep up on mentioning me... > - I'm spending too much time working on things in > pytest I'm somewhat interested in, but not really passionate about. Prioritising your own time to what you enjoy most is more then sensible. > I'd like to keep my maintainer/commit/whatever flag, and I might revisit > this in 1-2 years, where I might have more time on my hand for > open-source work. If there's consensus that I should drop it, I'm not > going to say no either, though. No need to remove a commit flag! (unless you intend to turn malicious ;)) All the best, Floris From kvas.it at gmail.com Tue Apr 17 17:22:38 2018 From: kvas.it at gmail.com (Vasily Kuznetsov) Date: Tue, 17 Apr 2018 21:22:38 +0000 Subject: [pytest-dev] Approval testing plugin for pytest Message-ID: Hi everyone! It would be nice to have an approval testing plugin for pytest, but it seems there's no perfect solution at the moment. A quick google search gave me this and this , but that's kind of far from ideal. What I would like to have is the ability to mark a test as approval test and then the framework would compare the return value of the test function to the approved output, produce error diffs, allow approving new output, etc. It would also play nice with fixtures, parametrization and other pytest features so that a separate approved result would be generated for each test item, stored in a file with some useful name and all of this would be transparent for the developer. A diffing tool for approving the outputs could be specified via configuration parameter or an environment variable. Does this sounds like something that you have seen? If not, I will probably try to build it (most likely from scratch, since not much of the functionality of ApprovalTest.Python would be useful here and I feel like I would struggle against their approval naming code) and would appreciate any advice. Cheers, Vasily -------------- next part -------------- An HTML attachment was scrubbed... URL: From variedthoughts at gmail.com Tue Apr 17 18:23:55 2018 From: variedthoughts at gmail.com (Brian Okken) Date: Tue, 17 Apr 2018 15:23:55 -0700 Subject: [pytest-dev] Approval testing plugin for pytest In-Reply-To: References: Message-ID: <1AA89E49-E6DE-46AC-A861-CFE0B9AA8CBB@gmail.com> I have seen something like what you are describing here: https://pypi.org/project/pytest-regtest/ - Brian Okken > On Apr 17, 2018, at 2:22 PM, Vasily Kuznetsov wrote: > > Hi everyone! > > It would be nice to have an approval testing plugin for pytest, but it seems there's no perfect solution at the moment. A quick google search gave me this and this, but that's kind of far from ideal. > > What I would like to have is the ability to mark a test as approval test and then the framework would compare the return value of the test function to the approved output, produce error diffs, allow approving new output, etc. It would also play nice with fixtures, parametrization and other pytest features so that a separate approved result would be generated for each test item, stored in a file with some useful name and all of this would be transparent for the developer. A diffing tool for approving the outputs could be specified via configuration parameter or an environment variable. > > Does this sounds like something that you have seen? If not, I will probably try to build it (most likely from scratch, since not much of the functionality of ApprovalTest.Python would be useful here and I feel like I would struggle against their approval naming code) and would appreciate any advice. > > Cheers, > Vasily > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From kvas.it at gmail.com Wed Apr 18 10:42:22 2018 From: kvas.it at gmail.com (Vasily Kuznetsov) Date: Wed, 18 Apr 2018 14:42:22 +0000 Subject: [pytest-dev] Approval testing plugin for pytest In-Reply-To: <1AA89E49-E6DE-46AC-A861-CFE0B9AA8CBB@gmail.com> References: <1AA89E49-E6DE-46AC-A861-CFE0B9AA8CBB@gmail.com> Message-ID: Oh, this looks pretty good! I will try it out. Thank you, Brian. Cheers, Vasily On Wed, Apr 18, 2018 at 12:23 AM Brian Okken wrote: > I have seen something like what you are describing here: > https://pypi.org/project/pytest-regtest/ > > - Brian Okken > > On Apr 17, 2018, at 2:22 PM, Vasily Kuznetsov wrote: > > Hi everyone! > > It would be nice to have an approval testing > plugin for pytest, but it seems there's no perfect solution at the moment. > A quick google search gave me this > and this > , but that's kind of > far from ideal. > > What I would like to have is the ability to mark a test as approval test > and then the framework would compare the return value of the test function > to the approved output, produce error diffs, allow approving new output, > etc. It would also play nice with fixtures, parametrization and other > pytest features so that a separate approved result would be generated for > each test item, stored in a file with some useful name and all of this > would be transparent for the developer. A diffing tool for approving the > outputs could be specified via configuration parameter or an environment > variable. > > Does this sounds like something that you have seen? If not, I will > probably try to build it (most likely from scratch, since not much of the > functionality of ApprovalTest.Python would be useful here and I feel like I > would struggle against their approval naming code) and would appreciate any > advice. > > Cheers, > Vasily > > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas+pytest at nicholaswilliams.net Wed Apr 18 14:19:08 2018 From: nicholas+pytest at nicholaswilliams.net (Nicholas Williams) Date: Wed, 18 Apr 2018 13:19:08 -0500 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. Message-ID: Hello, all. This is my first post to the mailing list, FWIW. I've been a PyTest user for some years, but now I'm working on writing a rather complex PyTest plugin to facilitate testing our service oriented architecture at $work. The purpose of the plugin is to collect additional tests (for which we are using `pytest_collection_modifyitems`) from a specialized file format that PyTest wouldn't normally understand. I've got it working beautifully, but I would love to hammer out some finer details to really make it a clean-running plugin. With that in mind, I'm looking for some suggestions / best practices / advice here on how best to handle these situations: Collection Errors The author of one of these specialized tests, which we call "test plans," could possibly do a few things wrong that cause more than just making their test fail. For example, they could get their file syntax incorrect, which results in a parsing error. At the moment, I catch the parsing error (we're using PyParsing to parse our test plans) and raise a different exception about the problem (and I've tried AssertionError), but everything I do results in a stack trace dumped with "INTERNALERROR>" and the immediate, abnormal termination of the PyTest process. This contrasts to, for an analog example, a syntax error in a regular Python test file, which instead results in a "ERROR collecting" output, while all other unaffected tests still run and the PyTest process terminates normally. What I'm looking for here is the "right way" to make my plugin report an "ERROR collecting" instead of causing an "INTERNALERROR>" when such a problem occurs collecting these specialized tests. Improving the Test Error Report When tests have errors or failures, PyTest prints out a report of those errors, and with each one includes an exhaustive stack trace. For our specialized tests, with some clever manipulation of the test data added to `items` in `pytest_collection_modifyitems`, I've managed to at least get it to display a meaningful name for the test that failed so that it's at least _possible_ for the developer to figure out which test, in which test file, failed. However, almost the entire stack trace is meaningless and full of plugin code instead of containing information useful to debugging the test. Additionally, I would _love_ to be able to add the actual test file and line number for the failure to the report. Would this be possible? Or, at least, would it be possible to filter the plugin code out of the stack trace so that it doesn't distract the developer from simply viewing the assertion error details? What approach should I take to achieve this? Thanks in advance for any help you can provide, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From ringo.de.smet at ontoforce.com Thu Apr 19 03:34:43 2018 From: ringo.de.smet at ontoforce.com (Ringo De Smet) Date: Thu, 19 Apr 2018 09:34:43 +0200 Subject: [pytest-dev] Unsubscribing from pytest-dev/pytest notifications In-Reply-To: <87tvs9zh63.fsf@powell.devork.be> References: <20180416055012.qufdcz3fiuzhy3cv@hooch.localdomain> <87tvs9zh63.fsf@powell.devork.be> Message-ID: Hello, On Tue, Apr 17, 2018 at 10:55 PM, Floris Bruynooghe wrote: > > > I really wish GitHub had a way to subscribe to issue labels, but until > > that happens, I need to unsubscribe from pytest-dev/pytest (which is > > what I just did) > > That's no problem at all. I've not been receiving email for github > activity in ages, and that blue dot on github is basically a permanent > feature. Volume is pretty high (to me), occasionally I try to look at > it but I mostly also rely on those of you who do keep up on mentioning > me... > I use https://octobox.io to follow up on all my Github activity (mentions, subscriptions, ...) It helped me, so I hope it can help you too. *Ringo De Smet* ringo.de.smet at ontoforce.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ringo.de.smet at ontoforce.com Thu Apr 19 07:20:08 2018 From: ringo.de.smet at ontoforce.com (Ringo De Smet) Date: Thu, 19 Apr 2018 13:20:08 +0200 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: References: Message-ID: Nicholas, Regarding the stack trace filtering, I have been bitten by this too in my custom plugin. You can read the solution here in the mail archives: https://mail.python.org/pipermail/pytest-dev/2018-March/004399.html If it is not clear enough, just reply and I might be able to help you forward. Ringo On Wed, Apr 18, 2018 at 8:19 PM, Nicholas Williams < nicholas+pytest at nicholaswilliams.net> wrote: > Hello, all. This is my first post to the mailing list, FWIW. > > I've been a PyTest user for some years, but now I'm working on writing a > rather complex PyTest plugin to facilitate testing our service oriented > architecture at $work. The purpose of the plugin is to collect additional > tests (for which we are using `pytest_collection_modifyitems`) from a > specialized file format that PyTest wouldn't normally understand. I've got > it working beautifully, but I would love to hammer out some finer details > to really make it a clean-running plugin. > > With that in mind, I'm looking for some suggestions / best practices / > advice here on how best to handle these situations: > > Collection Errors > The author of one of these specialized tests, which we call "test plans," > could possibly do a few things wrong that cause more than just making their > test fail. For example, they could get their file syntax incorrect, which > results in a parsing error. At the moment, I catch the parsing error (we're > using PyParsing to parse our test plans) and raise a different exception > about the problem (and I've tried AssertionError), but everything I do > results in a stack trace dumped with "INTERNALERROR>" and the immediate, > abnormal termination of the PyTest process. This contrasts to, for an > analog example, a syntax error in a regular Python test file, which instead > results in a "ERROR collecting" output, while all other unaffected tests > still run and the PyTest process terminates normally. What I'm looking for > here is the "right way" to make my plugin report an "ERROR collecting" > instead of causing an "INTERNALERROR>" when such a problem occurs > collecting these specialized tests. > > Improving the Test Error Report > When tests have errors or failures, PyTest prints out a report of those > errors, and with each one includes an exhaustive stack trace. For our > specialized tests, with some clever manipulation of the test data added to > `items` in `pytest_collection_modifyitems`, I've managed to at least get > it to display a meaningful name for the test that failed so that it's at > least _possible_ for the developer to figure out which test, in which test > file, failed. However, almost the entire stack trace is meaningless and > full of plugin code instead of containing information useful to debugging > the test. Additionally, I would _love_ to be able to add the actual test > file and line number for the failure to the report. Would this be possible? > Or, at least, would it be possible to filter the plugin code out of the > stack trace so that it doesn't distract the developer from simply viewing > the assertion error details? What approach should I take to achieve this? > > Thanks in advance for any help you can provide, > > Nick > > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > > -- *Ringo De Smet* ringo.de.smet at ontoforce.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ringo.de.smet at ontoforce.com Thu Apr 19 07:28:17 2018 From: ringo.de.smet at ontoforce.com (Ringo De Smet) Date: Thu, 19 Apr 2018 13:28:17 +0200 Subject: [pytest-dev] Adding plugins to pytest-dev In-Reply-To: <87wox5zi6x.fsf@powell.devork.be> References: <87wox5zi6x.fsf@powell.devork.be> Message-ID: Floris, Let me pass on how other communities manage Github teams, repositories and shared maintenance. Here is the example from the Chef (config management tool) community: https://github.com/sous-chefs/terraform-github-org They have everything scripted with Terraform[1] in a public read only repo so everybody can see who the maintainers are for each repo. Ringo [1] https://www.terraform.io On Tue, Apr 17, 2018 at 10:33 PM, Floris Bruynooghe wrote: > Hi all, > > While just looking at > https://github.com/pytest-dev/pytest/blob/master/ > CONTRIBUTING.rst#submitting-plugins-to-pytest-dev > I had some observations: > > 1. It says one of the objectives of having plugins under pytest-dev is > "Sharing some of the maintenance responsibility (in case a maintainer > no longer wishes to maintain a plugin)". > > I found this somewhat surprisingly worded. I thought it was more > "protect the pytest community in case a maintainer goes AWOL". I > think that's a significant difference, maintainers aren't supposed to > think they can move the plugin there if they no longer feel like > actively maintaining it. > > > 2. The other thing this says is "We recommend that each plugin has at > least three people who have the right to release to PyPI." I think > that's somewhat light and does not ensure a package can be released > if a maintainer does go AWOL. Could we change that to something like > "each plugin must have at least three people from > https://github.com/orgs/pytest-dev/teams/core/members who have the > right to release to PyPI" [0]? Practically we should probably > include a list of pypi usernames to that paragraph since there's not > mapping otherwise. > > > Any opinions on these? > > Cheers, > Floris > > > [0] Bonus points for whoever contributes a teams functionality to > warehouse! > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > -- *Ringo De Smet* ringo.de.smet at ontoforce.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide.moro at gmail.com Thu Apr 19 08:14:37 2018 From: davide.moro at gmail.com (davide moro) Date: Thu, 19 Apr 2018 14:14:37 +0200 Subject: [pytest-dev] new pytest-dev plugin proposal In-Reply-To: <87zi21ziqb.fsf@powell.devork.be> References: <87zi21ziqb.fsf@powell.devork.be> Message-ID: Hi Floris! 2018-04-17 22:22 GMT+02:00 Floris Bruynooghe : > Hello, > > On Thu 12 Apr 2018 at 10:57 +0000, Bruno Oliveira wrote: > > > Hi Davide, > > > > +1 to both. > > > > Now we need another +1 from some other core dev to start the transferring > > process. :) > > > > Cheers, > > Bruno. > > > > On Thu, Apr 12, 2018 at 4:46 AM davide moro > wrote: > > > >> Hi, > >> > >> I asked permission to my company for transferring the ownership of > github > >> repositories that might be interesting for the testing community to the > >> pytest-dev organization and they said yes. > > Cool! > > >> Talking before with Bruno we thought that the following one might be a > >> candidate: > >> * https://github.com/tierratelematics/pytest-play > > +1 on this > > Ok, thanks. What is the following step? Can I proceed with the ownership change? > >> There is a a cookiecutter scaffolding tool based on pytest providing a > >> working selenium setup, examples with pytest-bdd (it is not a pytest > >> plugin): > >> * https://github.com/tierratelematics/cookiecutter-qa > > I'm less sure about this one, it looks like pytest is not really the > core thing in this project. It's also not really a pytest plugin, but > yes I know about cookiecutter-pytest-plugin. But y'all think it fit's > just fine that ok with me. > > No problem, Let's freeze it if you prefer: I will be always available for a future ownership change! > Cheers, > Floris > Cheers, davide -- Linkedin: https://it.linkedin.com/in/davidemoro82 Twitter: https://twitter.com/davidemoro Blog: http://davidemoro.blogspot.it/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Thu Apr 19 14:54:32 2018 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 19 Apr 2018 20:54:32 +0200 Subject: [pytest-dev] new pytest-dev plugin proposal In-Reply-To: References: <87zi21ziqb.fsf@powell.devork.be> Message-ID: <87r2nbyql3.fsf@powell.devork.be> Hi Davide, On Thu 19 Apr 2018 at 14:14 +0200, davide moro wrote: > 2018-04-17 22:22 GMT+02:00 Floris Bruynooghe : >> On Thu 12 Apr 2018 at 10:57 +0000, Bruno Oliveira wrote: >> >> Talking before with Bruno we thought that the following one might be a >> >> candidate: >> >> * https://github.com/tierratelematics/pytest-play >> >> +1 on this >> > Ok, thanks. > > What is the following step? Can I proceed with the ownership change? Sure, follow https://github.com/pytest-dev/pytest/blob/master/CONTRIBUTING.rst#submitting-plugins-to-pytest-dev. For the github part you can probably either use Bruno (nicoddemus) or me (flub). For the pypi part adding the following users, copied from the list of people on pytest-faulthandler, is probably fine: anatoly, flub, hpk, nicoddemus, pfctdayelise, ronny, The_Compiler. Cheers, Floris From flub at devork.be Thu Apr 19 15:02:03 2018 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 19 Apr 2018 21:02:03 +0200 Subject: [pytest-dev] Adding plugins to pytest-dev In-Reply-To: References: <87wox5zi6x.fsf@powell.devork.be> Message-ID: <87o9ifyq8k.fsf@powell.devork.be> On Thu 19 Apr 2018 at 13:28 +0200, Ringo De Smet wrote: > Floris, > > Let me pass on how other communities manage Github teams, repositories and > shared maintenance. Here is the example from the Chef (config management > tool) community: > > https://github.com/sous-chefs/terraform-github-org > > They have everything scripted with Terraform[1] in a public read only repo > so everybody can see who the maintainers are for each repo. That's a nice bit of automation. From flub at devork.be Thu Apr 19 15:18:16 2018 From: flub at devork.be (Floris Bruynooghe) Date: Thu, 19 Apr 2018 21:18:16 +0200 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: References: Message-ID: <87lgdjyphj.fsf@powell.devork.be> Hi Nick, On Wed 18 Apr 2018 at 13:19 -0500, Nicholas Williams wrote: > Hello, all. This is my first post to the mailing list, FWIW. > > I've been a PyTest user for some years, but now I'm working on writing a > rather complex PyTest plugin to facilitate testing our service oriented > architecture at $work. The purpose of the plugin is to collect additional > tests (for which we are using `pytest_collection_modifyitems`) from a I think you'll fare better by integrating deeper into the collection mechanism instead of using pytest_collection_modifyitems. How are these test written? You probably should look at the pytest_collect_directory and pytest_collect_file hooks to do this. Unfortunately the documentation on these things is very thin. Finding existing code that does this is probably your best bet. The _pytest/python.py plugin is the canonical example though is very complicated. If you hunt other plugins you may find other examples. > specialized file format that PyTest wouldn't normally understand. I've got > it working beautifully, but I would love to hammer out some finer details > to really make it a clean-running plugin. > > With that in mind, I'm looking for some suggestions / best practices / > advice here on how best to handle these situations: > > Collection Errors > The author of one of these specialized tests, which we call "test plans," > could possibly do a few things wrong that cause more than just making their > test fail. For example, they could get their file syntax incorrect, which > results in a parsing error. At the moment, I catch the parsing error (we're > using PyParsing to parse our test plans) and raise a different exception > about the problem (and I've tried AssertionError), but everything I do > results in a stack trace dumped with "INTERNALERROR>" and the immediate, > abnormal termination of the PyTest process. This contrasts to, for an > analog example, a syntax error in a regular Python test file, which instead > results in a "ERROR collecting" output, while all other unaffected tests > still run and the PyTest process terminates normally. What I'm looking for > here is the "right way" to make my plugin report an "ERROR collecting" > instead of causing an "INTERNALERROR>" when such a problem occurs > collecting these specialized tests. I think these problems would go away if you join the collection mechanism "correctly". Not that "correctly" is necessarily easy or obvious... but the hints above might help you. > Improving the Test Error Report I think this was already answered. Cheers, Floris From oliver at bestwalter.de Thu Apr 19 15:17:32 2018 From: oliver at bestwalter.de (Oliver Bestwalter) Date: Thu, 19 Apr 2018 19:17:32 +0000 Subject: [pytest-dev] Adding plugins to pytest-dev In-Reply-To: <87o9ifyq8k.fsf@powell.devork.be> References: <87wox5zi6x.fsf@powell.devork.be> <87o9ifyq8k.fsf@powell.devork.be> Message-ID: Hi, Floris wrote: > That's a nice bit of automation. I like the idea that it works somehow, but after looking at the README it is still undistinguishable from magic to me. Floris wrote: > [0] Bonus points for whoever contributes a teams functionality to warehouse! This is why I think that this the definite way forward. Or ... if that won't happen just having a shared account sharing the secrets through keybase.io Cheers, Oliver -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas+pytest at nicholaswilliams.net Thu Apr 19 17:50:16 2018 From: nicholas+pytest at nicholaswilliams.net (Nicholas Williams) Date: Thu, 19 Apr 2018 16:50:16 -0500 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: <87lgdjyphj.fsf@powell.devork.be> References: <87lgdjyphj.fsf@powell.devork.be> Message-ID: The tests work this way: - There's a test class that inherits from our specialized test class (ServicePlanTestCase), which inherits from `unittest.TestCase` (for several reasons, this detail cannot be altered) - That test class can do setup and teardown like a normal test class, but it has a static attribute that points to a directory - That directory contains one or more files ending in a particular extension, and those files each contain one or more tests defined using a particular syntax Based on the hints you gave me, it sounds like I could do something like this: https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00cc21b79662459a8c9fafa87/_pytest/unittest.py#L14-L22 Only, instead of checking for inheritance from TestCase, I'd check for inheritance from our ServicePlanTestCase, and in that case I would return a new collector object that inherits from _pytest.python.Class, and write that new collector class to collect our tests. Am I barking up the right tree now? Thanks, Nick On Thu, Apr 19, 2018 at 2:18 PM, Floris Bruynooghe wrote: > Hi Nick, > > On Wed 18 Apr 2018 at 13:19 -0500, Nicholas Williams wrote: > > > Hello, all. This is my first post to the mailing list, FWIW. > > > > I've been a PyTest user for some years, but now I'm working on writing a > > rather complex PyTest plugin to facilitate testing our service oriented > > architecture at $work. The purpose of the plugin is to collect additional > > tests (for which we are using `pytest_collection_modifyitems`) from a > > I think you'll fare better by integrating deeper into the collection > mechanism instead of using pytest_collection_modifyitems. How are these > test written? You probably should look at the pytest_collect_directory > and pytest_collect_file hooks to do this. Unfortunately the > documentation on these things is very thin. Finding existing code that > does this is probably your best bet. The _pytest/python.py plugin is > the canonical example though is very complicated. If you hunt other > plugins you may find other examples. > > > specialized file format that PyTest wouldn't normally understand. I've > got > > it working beautifully, but I would love to hammer out some finer details > > to really make it a clean-running plugin. > > > > With that in mind, I'm looking for some suggestions / best practices / > > advice here on how best to handle these situations: > > > > Collection Errors > > The author of one of these specialized tests, which we call "test plans," > > could possibly do a few things wrong that cause more than just making > their > > test fail. For example, they could get their file syntax incorrect, which > > results in a parsing error. At the moment, I catch the parsing error > (we're > > using PyParsing to parse our test plans) and raise a different exception > > about the problem (and I've tried AssertionError), but everything I do > > results in a stack trace dumped with "INTERNALERROR>" and the immediate, > > abnormal termination of the PyTest process. This contrasts to, for an > > analog example, a syntax error in a regular Python test file, which > instead > > results in a "ERROR collecting" output, while all other unaffected tests > > still run and the PyTest process terminates normally. What I'm looking > for > > here is the "right way" to make my plugin report an "ERROR collecting" > > instead of causing an "INTERNALERROR>" when such a problem occurs > > collecting these specialized tests. > > I think these problems would go away if you join the collection > mechanism "correctly". Not that "correctly" is necessarily easy or > obvious... but the hints above might help you. > > > Improving the Test Error Report > > I think this was already answered. > > > Cheers, > Floris > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ringo.de.smet at ontoforce.com Fri Apr 20 02:26:13 2018 From: ringo.de.smet at ontoforce.com (Ringo De Smet) Date: Fri, 20 Apr 2018 08:26:13 +0200 Subject: [pytest-dev] Adding plugins to pytest-dev In-Reply-To: References: <87wox5zi6x.fsf@powell.devork.be> <87o9ifyq8k.fsf@powell.devork.be> Message-ID: Floris, Oliver, If you want, I can get you bootstrapped on this Terraform setup. I can say I almost know Terraform inside out. ;-) Ringo On Thu, Apr 19, 2018 at 9:17 PM, Oliver Bestwalter wrote: > Hi, > > Floris wrote: > > That's a nice bit of automation. > > I like the idea that it works somehow, but after looking at the README it > is still undistinguishable from magic to me. > > Floris wrote: > > [0] Bonus points for whoever contributes a teams functionality to > warehouse! > > This is why I think that this the definite way forward. Or ... if that > won't happen just having a shared account sharing the secrets through > keybase.io > > Cheers, > Oliver > -- *Ringo De Smet* ringo.de.smet at ontoforce.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Fri Apr 20 15:50:03 2018 From: flub at devork.be (Floris Bruynooghe) Date: Fri, 20 Apr 2018 21:50:03 +0200 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: References: <87lgdjyphj.fsf@powell.devork.be> Message-ID: <87in8lzmhg.fsf@powell.devork.be> On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote: > The tests work this way: > > - There's a test class that inherits from our specialized test class > (ServicePlanTestCase), which inherits from `unittest.TestCase` (for several > reasons, this detail cannot be altered) > - That test class can do setup and teardown like a normal test class, but > it has a static attribute that points to a directory > - That directory contains one or more files ending in a particular > extension, and those files each contain one or more tests defined using a > particular syntax I'm not even going to ask how this all came to be this way... but thanks for giving enough context. > Based on the hints you gave me, it sounds like I could do something like > this: > https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00cc21b79662459a8c9fafa87/_pytest/unittest.py#L14-L22 > > Only, instead of checking for inheritance from TestCase, I'd check for > inheritance from our ServicePlanTestCase, and in that case I would return a > new collector object that inherits from _pytest.python.Class, and write > that new collector class to collect our tests. Am I barking up the right > tree now? pytest_pycollect_makeitem is supposed to create a single item, while from what you describe you still have two collections nested. You should probably create a custom collection node for your class, which then creates a different kind of collection node for each file which in turn creates the actual test nodes in it's .collect(). From oliver at bestwalter.de Fri Apr 20 13:52:54 2018 From: oliver at bestwalter.de (Oliver Bestwalter) Date: Fri, 20 Apr 2018 17:52:54 +0000 Subject: [pytest-dev] Adding plugins to pytest-dev In-Reply-To: References: <87wox5zi6x.fsf@powell.devork.be> <87o9ifyq8k.fsf@powell.devork.be> Message-ID: Thank you Ringo, I have no immediate plans to implement something like this but I would definitely appreciate some further docs/explanations/blog that unravels how it is accomplished. Cheers, Oliver On Fri 20. Apr 2018 at 08:26, Ringo De Smet wrote: > Floris, Oliver, > > If you want, I can get you bootstrapped on this Terraform setup. I can say > I almost know Terraform inside out. ;-) > > Ringo > > On Thu, Apr 19, 2018 at 9:17 PM, Oliver Bestwalter > wrote: > >> Hi, >> >> Floris wrote: >> > That's a nice bit of automation. >> >> I like the idea that it works somehow, but after looking at the README it >> is still undistinguishable from magic to me. >> >> Floris wrote: >> > [0] Bonus points for whoever contributes a teams functionality to >> warehouse! >> >> This is why I think that this the definite way forward. Or ... if that >> won't happen just having a shared account sharing the secrets through >> keybase.io >> >> Cheers, >> Oliver >> > > > > -- > *Ringo De Smet* > ringo.de.smet at ontoforce.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Sat Apr 21 04:50:55 2018 From: flub at devork.be (Floris Bruynooghe) Date: Sat, 21 Apr 2018 10:50:55 +0200 Subject: [pytest-dev] Adding plugins to pytest-dev In-Reply-To: References: <87wox5zi6x.fsf@powell.devork.be> <87o9ifyq8k.fsf@powell.devork.be> Message-ID: <87d0ytymc0.fsf@powell.devork.be> Hi Ringo, On Fri 20 Apr 2018 at 08:26 +0200, Ringo De Smet wrote: > If you want, I can get you bootstrapped on this Terraform setup. I can say > I almost know Terraform inside out. ;-) Thanks for the offer to help with the project! One downside I see with doing this is that it's a fair amount of complexity which mostly only you will understand and be able to maintain. I'm not sure the current maintenance overhead of doing things by hand warrants taking such steps yet. But let's maybe check what the requirements would be before jumping to such conclusions, on top of my head: - have groups of users like pytest-dev-core pytest-plugin-authors etc - configure users/groups/repos on github - configure users/groups/repos on bitbucket - configure users/groups/projects on pypi - whatever else I forgot Would it be able to help with all of these? Cheers, Floris > > Ringo > > On Thu, Apr 19, 2018 at 9:17 PM, Oliver Bestwalter > wrote: > >> Hi, >> >> Floris wrote: >> > That's a nice bit of automation. >> >> I like the idea that it works somehow, but after looking at the README it >> is still undistinguishable from magic to me. >> >> Floris wrote: >> > [0] Bonus points for whoever contributes a teams functionality to >> warehouse! >> >> This is why I think that this the definite way forward. Or ... if that >> won't happen just having a shared account sharing the secrets through >> keybase.io >> >> Cheers, >> Oliver >> From ringo.de.smet at ontoforce.com Sat Apr 21 14:25:11 2018 From: ringo.de.smet at ontoforce.com (Ringo De Smet) Date: Sat, 21 Apr 2018 20:25:11 +0200 Subject: [pytest-dev] Adding plugins to pytest-dev In-Reply-To: <87d0ytymc0.fsf@powell.devork.be> References: <87wox5zi6x.fsf@powell.devork.be> <87o9ifyq8k.fsf@powell.devork.be> <87d0ytymc0.fsf@powell.devork.be> Message-ID: Floris, On Sat, Apr 21, 2018 at 10:50 AM, Floris Bruynooghe wrote: > > One downside I see with doing this is that it's a fair amount of > complexity which mostly only you will understand and be able to > maintain. I'm not sure the current maintenance overhead of doing things > by hand warrants taking such steps yet. But let's maybe check what the > requirements would be before jumping to such conclusions, on top of my > head: > > - have groups of users like pytest-dev-core pytest-plugin-authors etc > - configure users/groups/repos on github > - configure users/groups/repos on bitbucket > - configure users/groups/projects on pypi > - whatever else I forgot > > Would it be able to help with all of these? > No, currently it would only help with setting up users/groups/repos on your Github organization. *Ringo De Smet* ringo.de.smet at ontoforce.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide.moro at gmail.com Mon Apr 23 18:08:05 2018 From: davide.moro at gmail.com (davide moro) Date: Tue, 24 Apr 2018 00:08:05 +0200 Subject: [pytest-dev] new pytest-dev plugin proposal In-Reply-To: <87r2nbyql3.fsf@powell.devork.be> References: <87zi21ziqb.fsf@powell.devork.be> <87r2nbyql3.fsf@powell.devork.be> Message-ID: Done: * https://github.com/pytest-dev/pytest-play Cheers, davide 2018-04-19 20:54 GMT+02:00 Floris Bruynooghe : > Hi Davide, > > On Thu 19 Apr 2018 at 14:14 +0200, davide moro wrote: > > 2018-04-17 22:22 GMT+02:00 Floris Bruynooghe : > >> On Thu 12 Apr 2018 at 10:57 +0000, Bruno Oliveira wrote: > >> >> Talking before with Bruno we thought that the following one might be > a > >> >> candidate: > >> >> * https://github.com/tierratelematics/pytest-play > >> > >> +1 on this > >> > > Ok, thanks. > > > > What is the following step? Can I proceed with the ownership change? > > Sure, follow > https://github.com/pytest-dev/pytest/blob/master/ > CONTRIBUTING.rst#submitting-plugins-to-pytest-dev. > For the github part you can probably either use Bruno (nicoddemus) or me > (flub). For the pypi part adding the following users, copied from the > list of people on pytest-faulthandler, is probably fine: anatoly, flub, > hpk, nicoddemus, pfctdayelise, ronny, The_Compiler. > > > > Cheers, > Floris > -- Linkedin: https://it.linkedin.com/in/davidemoro82 Twitter: https://twitter.com/davidemoro Blog: http://davidemoro.blogspot.it/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas+pytest at nicholaswilliams.net Mon Apr 23 18:42:44 2018 From: nicholas+pytest at nicholaswilliams.net (Nicholas Williams) Date: Mon, 23 Apr 2018 17:42:44 -0500 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: <87in8lzmhg.fsf@powell.devork.be> References: <87lgdjyphj.fsf@powell.devork.be> <87in8lzmhg.fsf@powell.devork.be> Message-ID: Floris, your advice was invaluable. We have created a really fantastic PyTest plugin for our service tests. However, I wanted to share it with you, because it seems to have introduced some type of weird conflict with the PyTest Coverage plugin, and I was hoping maybe you had some insights as to what the problem might be: Normally, when I run PyTest tests with the PyTest Coverage plugin, everything works as expected in regards to the coverage calculation and the coverage reports. And, with this change to our project, everything STILL works properly IF AND ONLY IF I run tests with `coverage run --source=pysoa setup.py test` followed by `coverage report` (so, without the PyTest Coverage plugin). However, with this change to our project, something VERY weird happens to the coverage report if I rely on the PyTest Coverage plugin. Now, ONLY LINES OF CODE WITHIN FUNCTIONS AND METHODS are marked as covered. The following types of code, which were previously marked as covered (and are clearly covered), are now marked as uncovered (again, only when using the PyTest Coverage plugin): - module-level imports - module-level code that executes on import - module docstrings - module-level class definitions (class ClassName(...):), or class definitions within other class definitions - code that executes within a class on definition - class docstrings - function definition lines (def func_name(...):), but not the code within the functions, which is marked as covered - method definition lines within classes (def method_name(...):), but not the code within the functions, which is marked as covered While the entire pull request is large and daunting, the most relevant part is the PyTest plugin itself, which is contained in a single file "pytest_plugin.py" that you can search for on the page. Here it is: https://github.com/eventbrite/pysoa/pull/87/files Unfortunately, I did not notice this problem with coverage until the bulk of the code was written and tested, so determining what part of the code caused it by removing bits of code is ... impracticable. I'm hoping one of the pytest-cov experts in here will have some idea what the heck is going on and can point me in the right direction. Perhaps I'm doing something wrong, but this smells very strongly of a pytest-cov bug somehow. Thanks, Nick On Fri, Apr 20, 2018 at 2:50 PM, Floris Bruynooghe wrote: > On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote: > > > The tests work this way: > > > > - There's a test class that inherits from our specialized test class > > (ServicePlanTestCase), which inherits from `unittest.TestCase` (for > several > > reasons, this detail cannot be altered) > > - That test class can do setup and teardown like a normal test class, but > > it has a static attribute that points to a directory > > - That directory contains one or more files ending in a particular > > extension, and those files each contain one or more tests defined using a > > particular syntax > > I'm not even going to ask how this all came to be this way... but thanks > for giving enough context. > > > Based on the hints you gave me, it sounds like I could do something like > > this: > > https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00cc21b79662459 > a8c9fafa87/_pytest/unittest.py#L14-L22 > > > > Only, instead of checking for inheritance from TestCase, I'd check for > > inheritance from our ServicePlanTestCase, and in that case I would > return a > > new collector object that inherits from _pytest.python.Class, and write > > that new collector class to collect our tests. Am I barking up the right > > tree now? > > pytest_pycollect_makeitem is supposed to create a single item, > while from what you describe you still have two collections nested. You > should probably create a custom collection node for your class, which > then creates a different kind of collection node for each file which in > turn creates the actual test nodes in it's .collect(). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas+pytest at nicholaswilliams.net Tue Apr 24 08:03:28 2018 From: nicholas+pytest at nicholaswilliams.net (Nicholas Williams) Date: Tue, 24 Apr 2018 07:03:28 -0500 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: References: <87lgdjyphj.fsf@powell.devork.be> <87in8lzmhg.fsf@powell.devork.be> Message-ID: Wow. After carefully reading the pytest-cov plugin (only) code and the PyTest code that powers loading plugins, I figured it out. The plugin was getting loaded before pytest-cov started the Coverage analyzer. Since the plugin had module-level imports from other parts of our project, that meant that most of the project was imported (and, so, module-level imports and code and class definitions had already executed) before Coverage started. The fix was actually very simple: I moved the plugin into a different module and then moved its top-level imports for the same project to happen at the last second. Viola. Coverage is fixed. Now just the plugin module itself reports wonky coverage numbers, but I can live with that. Thanks again for your help! Nick On Mon, Apr 23, 2018 at 5:42 PM, Nicholas Williams < nicholas+pytest at nicholaswilliams.net> wrote: > Floris, your advice was invaluable. We have created a really fantastic > PyTest plugin for our service tests. However, I wanted to share it with > you, because it seems to have introduced some type of weird conflict with > the PyTest Coverage plugin, and I was hoping maybe you had some insights as > to what the problem might be: > > Normally, when I run PyTest tests with the PyTest Coverage plugin, > everything works as expected in regards to the coverage calculation and the > coverage reports. And, with this change to our project, everything STILL > works properly IF AND ONLY IF I run tests with `coverage run --source=pysoa > setup.py test` followed by `coverage report` (so, without the PyTest > Coverage plugin). However, with this change to our project, something VERY > weird happens to the coverage report if I rely on the PyTest Coverage > plugin. Now, ONLY LINES OF CODE WITHIN FUNCTIONS AND METHODS are marked as > covered. The following types of code, which were previously marked as > covered (and are clearly covered), are now marked as uncovered (again, only > when using the PyTest Coverage plugin): > > - module-level imports > - module-level code that executes on import > - module docstrings > - module-level class definitions (class ClassName(...):), or class > definitions within other class definitions > - code that executes within a class on definition > - class docstrings > - function definition lines (def func_name(...):), but not the code within > the functions, which is marked as covered > - method definition lines within classes (def method_name(...):), but not > the code within the functions, which is marked as covered > > While the entire pull request is large and daunting, the most relevant > part is the PyTest plugin itself, which is contained in a single file > "pytest_plugin.py" that you can search for on the page. Here it is: > https://github.com/eventbrite/pysoa/pull/87/files > > Unfortunately, I did not notice this problem with coverage until the bulk > of the code was written and tested, so determining what part of the code > caused it by removing bits of code is ... impracticable. I'm hoping one of > the pytest-cov experts in here will have some idea what the heck is going > on and can point me in the right direction. Perhaps I'm doing something > wrong, but this smells very strongly of a pytest-cov bug somehow. > > Thanks, > > Nick > > > On Fri, Apr 20, 2018 at 2:50 PM, Floris Bruynooghe wrote: > >> On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote: >> >> > The tests work this way: >> > >> > - There's a test class that inherits from our specialized test class >> > (ServicePlanTestCase), which inherits from `unittest.TestCase` (for >> several >> > reasons, this detail cannot be altered) >> > - That test class can do setup and teardown like a normal test class, >> but >> > it has a static attribute that points to a directory >> > - That directory contains one or more files ending in a particular >> > extension, and those files each contain one or more tests defined using >> a >> > particular syntax >> >> I'm not even going to ask how this all came to be this way... but thanks >> for giving enough context. >> >> > Based on the hints you gave me, it sounds like I could do something like >> > this: >> > https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00c >> c21b79662459a8c9fafa87/_pytest/unittest.py#L14-L22 >> > >> > Only, instead of checking for inheritance from TestCase, I'd check for >> > inheritance from our ServicePlanTestCase, and in that case I would >> return a >> > new collector object that inherits from _pytest.python.Class, and write >> > that new collector class to collect our tests. Am I barking up the right >> > tree now? >> >> pytest_pycollect_makeitem is supposed to create a single item, >> while from what you describe you still have two collections nested. You >> should probably create a custom collection node for your class, which >> then creates a different kind of collection node for each file which in >> turn creates the actual test nodes in it's .collect(). >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicoddemus at gmail.com Tue Apr 24 18:32:41 2018 From: nicoddemus at gmail.com (Bruno Oliveira) Date: Tue, 24 Apr 2018 22:32:41 +0000 Subject: [pytest-dev] pytest 3.5.1 released Message-ID: pytest 3.5.1 has just been released to PyPI. This is a bug-fix release, being a drop-in replacement. To upgrade:: pip install --upgrade pytest The full changelog is available at http://doc.pytest.org/en/latest/changelog.html. Thanks to all who contributed to this release, among them: * Brian Maissy * Bruno Oliveira * Darren Burns * David Chudzicki * Floris Bruynooghe * Holger Kohr * Irmen de Jong * Jeffrey Rackauckas * Rachel Kogan * Ronny Pfannschmidt * Stefan Scherfke * Tim Strazny * ????? ???????? Happy testing, The pytest Development Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Fri Apr 27 16:06:05 2018 From: flub at devork.be (Floris Bruynooghe) Date: Fri, 27 Apr 2018 22:06:05 +0200 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: References: <87lgdjyphj.fsf@powell.devork.be> <87in8lzmhg.fsf@powell.devork.be> Message-ID: <877eosza6q.fsf@powell.devork.be> Hi Nick, On Tue 24 Apr 2018 at 07:03 -0500, Nicholas Williams wrote: > Wow. After carefully reading the pytest-cov plugin (only) code and the > PyTest code that powers loading plugins, I figured it out. The plugin was > getting loaded before pytest-cov started the Coverage analyzer. Since the > plugin had module-level imports from other parts of our project, that meant > that most of the project was imported (and, so, module-level imports and > code and class definitions had already executed) before Coverage > started. Nice you figured this annoying issue out while I was ignoring my mail for a while... :) This is a bit brittle but unavoidable unfortunately. > Thanks again for your help! Glad my vague pointers proved useful! Cheers, Floris From nicholas+pytest at nicholaswilliams.net Fri Apr 27 16:15:51 2018 From: nicholas+pytest at nicholaswilliams.net (Nicholas Williams) Date: Fri, 27 Apr 2018 15:15:51 -0500 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: <877eosza6q.fsf@powell.devork.be> References: <87lgdjyphj.fsf@powell.devork.be> <87in8lzmhg.fsf@powell.devork.be> <877eosza6q.fsf@powell.devork.be> Message-ID: One thing that concerned me in this whole process is that the vast majority of PyTest code appears to be behind a private API named `_pytest`. I was unable to implement this plugin without accessing these APIs. I could find no way around it. I minimized the imports as much as I could, even copying smaller sections of code instead of importing, but I still had to make these imports: from _pytest.unittest import ( TestCaseFunction, UnitTestCase, ) from _pytest._code.code import TracebackEntry from _pytest._code.source import Source from _pytest.mark import MARK_GEN How much are we going to be able to rely on these staying fairly stable? I'd feel much better if these were public APIs. :-/ Thanks, Nick On Fri, Apr 27, 2018 at 3:06 PM, Floris Bruynooghe wrote: > Hi Nick, > > On Tue 24 Apr 2018 at 07:03 -0500, Nicholas Williams wrote: > > > Wow. After carefully reading the pytest-cov plugin (only) code and the > > PyTest code that powers loading plugins, I figured it out. The plugin was > > getting loaded before pytest-cov started the Coverage analyzer. Since the > > plugin had module-level imports from other parts of our project, that > meant > > that most of the project was imported (and, so, module-level imports and > > code and class definitions had already executed) before Coverage > > started. > > Nice you figured this annoying issue out while I was ignoring my mail > for a while... :) This is a bit brittle but unavoidable unfortunately. > > > Thanks again for your help! > > Glad my vague pointers proved useful! > > Cheers, > Floris > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Sat Apr 28 12:53:44 2018 From: flub at devork.be (Floris Bruynooghe) Date: Sat, 28 Apr 2018 18:53:44 +0200 Subject: [pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc. In-Reply-To: References: <87lgdjyphj.fsf@powell.devork.be> <87in8lzmhg.fsf@powell.devork.be> <877eosza6q.fsf@powell.devork.be> Message-ID: <874ljvz2zr.fsf@powell.devork.be> On Fri 27 Apr 2018 at 15:15 -0500, Nicholas Williams wrote: > One thing that concerned me in this whole process is that the vast majority > of PyTest code appears to be behind a private API named `_pytest`. I was > unable to implement this plugin without accessing these APIs. I could find > no way around it. I minimized the imports as much as I could, even copying > smaller sections of code instead of importing, but I still had to make > these imports: > > from _pytest.unittest import ( > TestCaseFunction, > UnitTestCase, > ) I think the intention was the public API would be pytest.File, pytest.Function, pytest.Collector, pytest.Instance and pytest.Item (listed here in no particular order). Your use case seems to want to customise the existing unittest support which IIRC was never imagined this would happen. Hence you're either stuck re-creating lots of things for the unittest support using the public APIs I've listed or importing the private things I guess. So maybe you have a valid case for wanting pytest to expose a few more of the collecting semantics. Would be best to discuss and examine this in an issue/feature request I guess (if you do so please mention me in the issue as otherwise I'll probably miss it) > from _pytest._code.code import TracebackEntry > from _pytest._code.source import Source > from _pytest.mark import MARK_GEN These are maybe somewhat more troubling, I'd have to look into these carefully. > How much are we going to be able to rely on these staying fairly stable? > I'd feel much better if these were public APIs. :-/ As I kind of suggest above, the second lot seems a lot scarier to me and more likely to easily break. Regards, Floris From s.apostolico at gmail.com Sat Apr 28 13:17:52 2018 From: s.apostolico at gmail.com (Stefano) Date: Sat, 28 Apr 2018 19:17:52 +0200 Subject: [pytest-dev] request to add pytest-echo to pytest-dev plugins Message-ID: Hi, I did a very small plugin (pytest-echo), let me know if it can be added to the pytest-dev organization. Code can be found at https://github.com/saxix/pytest-echo Best, Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicoddemus at gmail.com Sat Apr 28 20:07:48 2018 From: nicoddemus at gmail.com (Bruno Oliveira) Date: Sun, 29 Apr 2018 00:07:48 +0000 Subject: [pytest-dev] request to add pytest-echo to pytest-dev plugins In-Reply-To: References: Message-ID: Hi Stefano, The plugin meets all requirements so +1 from me. :) Cheers On Sat, Apr 28, 2018 at 2:18 PM Stefano wrote: > Hi, > > I did a very small plugin (pytest-echo), let me know if it can be added > to the pytest-dev organization. > > Code can be found at https://github.com/saxix/pytest-echo > > > Best, > Stefano > _______________________________________________ > pytest-dev mailing list > pytest-dev at python.org > https://mail.python.org/mailman/listinfo/pytest-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.apostolico at gmail.com Sun Apr 29 12:36:44 2018 From: s.apostolico at gmail.com (Stefano) Date: Sun, 29 Apr 2018 18:36:44 +0200 Subject: [pytest-dev] request to add pytest-echo to pytest-dev plugins In-Reply-To: References: Message-ID: Dear Bruno, thanks for your reply. I have to wait one more positive feedback before starting to transfer the ownership, is this correct ? Cheers, Stefano 2018-04-29 2:07 GMT+02:00 Bruno Oliveira : > Hi Stefano, > > The plugin meets all requirements so +1 from me. :) > > Cheers > > On Sat, Apr 28, 2018 at 2:18 PM Stefano wrote: > >> Hi, >> >> I did a very small plugin (pytest-echo), let me know if it can be added >> to the pytest-dev organization. >> >> Code can be found at https://github.com/saxix/pytest-echo >> >> >> Best, >> Stefano >> _______________________________________________ >> pytest-dev mailing list >> pytest-dev at python.org >> https://mail.python.org/mailman/listinfo/pytest-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicoddemus at gmail.com Sun Apr 29 12:39:35 2018 From: nicoddemus at gmail.com (Bruno Oliveira) Date: Sun, 29 Apr 2018 16:39:35 +0000 Subject: [pytest-dev] request to add pytest-echo to pytest-dev plugins In-Reply-To: References: Message-ID: Hi Stefano, Yep according to our rules we still need one more +1. Cheers, On Sun, Apr 29, 2018 at 1:37 PM Stefano wrote: > Dear Bruno, > > thanks for your reply. > > I have to wait one more positive feedback before starting to transfer the > ownership, is this correct ? > > > Cheers, > Stefano > > > 2018-04-29 2:07 GMT+02:00 Bruno Oliveira : > >> Hi Stefano, >> >> The plugin meets all requirements so +1 from me. :) >> >> Cheers >> >> On Sat, Apr 28, 2018 at 2:18 PM Stefano wrote: >> >>> Hi, >>> >>> I did a very small plugin (pytest-echo), let me know if it can be added >>> to the pytest-dev organization. >>> >>> Code can be found at https://github.com/saxix/pytest-echo >>> >>> >>> Best, >>> Stefano >>> _______________________________________________ >>> pytest-dev mailing list >>> pytest-dev at python.org >>> https://mail.python.org/mailman/listinfo/pytest-dev >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: