From memedough at gmail.com Mon Aug 2 15:45:10 2010 From: memedough at gmail.com (meme dough) Date: Mon, 2 Aug 2010 23:45:10 +1000 Subject: [py-dev] pytest-cov Message-ID: Hi, I decided not to wait for coverage source option to be released. I will release again when it is available. pytest-cov now has slimmer command line interface which is pleasant when doing py.test -h It should measure sub processes out of the box (provided I got the pth file installed in the correct location, works with what I have available for testing). It will do the sub processes for distributed testing too. I will be putting it through it's paces in more depth. A doc update on the site would probably be good. Give it a try if anyone interested and let me know how it goes. :) From holger at merlinux.eu Mon Aug 2 16:05:43 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 2 Aug 2010 16:05:43 +0200 Subject: [py-dev] pytest-cov In-Reply-To: References: Message-ID: <20100802140543.GP1914@trillke.net> Hi Meme, On Mon, Aug 02, 2010 at 23:45 +1000, meme dough wrote: > Hi, > > I decided not to wait for coverage source option to be released. I > will release again when it is available. good idea :) > pytest-cov now has slimmer command line interface which is pleasant > when doing py.test -h very nice! > It should measure sub processes out of the box (provided I got the pth > file installed in the correct location, works with what I have > available for testing). It will do the sub processes for distributed > testing too. Hum, how does the "out-of-the-box" work exactly? It won't always be enabled just because pytest-cov is installed, will it? (if so, i'd rather suggest to do an indirection and only perform coverage if a certain environment varialbe like PYTEST_COV_AUTOSUB=1 or so it st). > I will be putting it through it's paces in more depth. > > A doc update on the site would probably be good. > > Give it a try if anyone interested and let me know how it goes. I tried it, worked fine (and i really like the slim interface!). Notes: * --cov=path ... the help string should say what path, file-system or python import-path? * when i ran "py.test --cov=py._code" it worked fine, i.e. only showed results for the py._code subpackage. When i mistype the path, e.g. "py.test --cov=py._code2" it will give me *all* results which is wrong: i'd expect "no coverage data for import path "py._code2" found or something like this. cheers, holger From holger at merlinux.eu Tue Aug 3 10:14:28 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 Aug 2010 10:14:28 +0200 Subject: [py-dev] pytest-cov In-Reply-To: <20100802140543.GP1914@trillke.net> References: <20100802140543.GP1914@trillke.net> Message-ID: <20100803081428.GQ1914@trillke.net> On Mon, Aug 02, 2010 at 16:05 +0200, holger krekel wrote: > Hum, how does the "out-of-the-box" work exactly? > It won't always be enabled just because pytest-cov is installed, will it? > (if so, i'd rather suggest to do an indirection and only perform coverage > if a certain environment varialbe like PYTEST_COV_AUTOSUB=1 or so it st). FYI i think pytest-cov begins to work better than pytest-figleaf in some cases! :) However, i consider it crucial that installing pytest-cov has no side effects by default. With the above suggestion we could make py.test set the environment variable when a coverage-run is selected and otherwise avoid any side effects, particularly when py.test is not even run. IOW, i'd be very happy with a pytest-cov-1.1 streamlining this issue so it's safe to point people to it :) best, holger > > > I will be putting it through it's paces in more depth. > > > > A doc update on the site would probably be good. > > > > Give it a try if anyone interested and let me know how it goes. > > I tried it, worked fine (and i really like the slim interface!). > Notes: > > * --cov=path ... the help string should say what path, file-system or > python import-path? > > * when i ran "py.test --cov=py._code" it worked fine, i.e. only showed > results for the py._code subpackage. When i mistype the path, > e.g. "py.test --cov=py._code2" it will give me *all* results which > is wrong: i'd expect "no coverage data for import path "py._code2" found > or something like this. > > cheers, > holger > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- From memedough at gmail.com Tue Aug 3 13:57:21 2010 From: memedough at gmail.com (meme dough) Date: Tue, 3 Aug 2010 21:57:21 +1000 Subject: [py-dev] pytest-cov In-Reply-To: <20100803081428.GQ1914@trillke.net> References: <20100802140543.GP1914@trillke.net> <20100803081428.GQ1914@trillke.net> Message-ID: Hi, On 3 August 2010 18:14, holger krekel wrote: > On Mon, Aug 02, 2010 at 16:05 +0200, holger krekel wrote: >> Hum, how does the "out-of-the-box" work exactly? >> It won't always be enabled just because pytest-cov is installed, will it? >> (if so, i'd rather suggest to do an indirection and only perform coverage >> if a certain environment varialbe like PYTEST_COV_AUTOSUB=1 or so it st). > > FYI i think pytest-cov begins to work better than pytest-figleaf > in some cases! :) > > However, i consider it crucial that installing pytest-cov has no side effects > by default. ?With the above suggestion we could make py.test set the environment > variable when a coverage-run is selected and otherwise avoid any side effects, > particularly when py.test is not even run. ?IOW, i'd be very happy with a > pytest-cov-1.1 streamlining this issue so it's safe to point people to it :) When pytest-cov installed (well really cov-core) an init_cov_core.pth will be created. During python site initialisation this will cause coverage to start only if appropriate. From the doc string: Activate coverage at python startup if appropriate. The python site initialisation will ensure that anything we import will be removed and not visible at the end of python startup. However we minimise all work by putting these init actions in this separate module and only importing what is needed when needed. For normal python startup when coverage should not be activated we only import os, look for one env var and get out. For python startup when an ancestor process has set the env indicating that code coverage is being collected we activate coverage based on info passed via env vars. So I completely agree with you. One issue is creating the pth file during installation. It works for me with what I have to test with. If it can't find site-packages / dist-packages then it outputs a message describing what file to create and content to go in it. Another issue is uninstalling. I added a big note indicating that the pth file must be uninstalled manually. >> * --cov=path ... the help string should say what path, file-system or >> ? python import-path? I was tying to be consistent with other pytest options and path looked the best as used by others. Under hood it is currently morf (modules or files coverage speak) but will change to source option which is a root of a source tree so I think path is the good like following. Suggestions? --junitxml=path --ignore=path --genscript=path >> * when i ran "py.test --cov=py._code" it worked fine, i.e. only showed >> ? results for the py._code subpackage. ?When i mistype the path, >> ? e.g. "py.test --cov=py._code2" it will give me *all* results which >> ? is wrong: i'd expect "no coverage data for import path "py._code2" found >> ? or something like this. If all --cov options don't give something valid then everything collected is dumped (and everything was collect unless you use omit option in config file and omit is changing a lot with new source / include / omit). This is where source option comes in. With source option specify the roots of trees then limited to those roots since measurement itself is not done outside of those trees. :) From holger at merlinux.eu Tue Aug 3 14:17:00 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 Aug 2010 14:17:00 +0200 Subject: [py-dev] pytest-cov In-Reply-To: References: <20100802140543.GP1914@trillke.net> <20100803081428.GQ1914@trillke.net> Message-ID: <20100803121700.GR1914@trillke.net> Hi Meme, On Tue, Aug 03, 2010 at 21:57 +1000, meme dough wrote: > On 3 August 2010 18:14, holger krekel wrote: > > On Mon, Aug 02, 2010 at 16:05 +0200, holger krekel wrote: > >> Hum, how does the "out-of-the-box" work exactly? > >> It won't always be enabled just because pytest-cov is installed, will it? > >> (if so, i'd rather suggest to do an indirection and only perform coverage > >> if a certain environment varialbe like PYTEST_COV_AUTOSUB=1 or so it st). > > > > FYI i think pytest-cov begins to work better than pytest-figleaf > > in some cases! :) > > > > However, i consider it crucial that installing pytest-cov has no side effects > > by default. ?With the above suggestion we could make py.test set the environment > > variable when a coverage-run is selected and otherwise avoid any side effects, > > particularly when py.test is not even run. ?IOW, i'd be very happy with a > > pytest-cov-1.1 streamlining this issue so it's safe to point people to it :) > > When pytest-cov installed (well really cov-core) an init_cov_core.pth > will be created. During python site initialisation this will cause > coverage to start only if appropriate. From the doc string: > > Activate coverage at python startup if appropriate. > > The python site initialisation will ensure that anything we import > will be removed and not visible at the end of python startup. However > we minimise all work by putting these init actions in this separate > module and only importing what is needed when needed. > > For normal python startup when coverage should not be activated we > only import os, look for one env var and get out. > > For python startup when an ancestor process has set the env indicating > that code coverage is being collected we activate coverage based on > info passed via env vars. If i understand correctly you are describing the situation as is. Sorry, i didn't look at the content of cov_core_init.py which would have revealed the mechanics that you describe. > So I completely agree with you. > > One issue is creating the pth file during installation. It works for > me with what I have to test with. If it can't find site-packages / > dist-packages then it outputs a message describing what file to create > and content to go in it. > > Another issue is uninstalling. I added a big note indicating that the > pth file must be uninstalled manually. I guess somehow "pip" should be better able to uninstall. But ok, cov_core_init.py is hopefully not causing trouble so it's not so bad. > >> * --cov=path ... the help string should say what path, file-system or > >> ? python import-path? > > I was tying to be consistent with other pytest options and path looked > the best as used by others. Under hood it is currently morf (modules > or files coverage speak) but will change to source option which is a > root of a source tree so I think path is the good like following. > Suggestions? > > --junitxml=path > --ignore=path > --genscript=path these are all file system paths. > >> * when i ran "py.test --cov=py._code" it worked fine, i.e. only showed > >> ? results for the py._code subpackage. ?When i mistype the path, > >> ? e.g. "py.test --cov=py._code2" it will give me *all* results which > >> ? is wrong: i'd expect "no coverage data for import path "py._code2" found > >> ? or something like this. > > If all --cov options don't give something valid then everything > collected is dumped (and everything was collect unless you use omit > option in config file and omit is changing a lot with new source / > include / omit). This remains then the only issue i guess: rather give information that the specified "--cov=XYZ" restriction was used and turn up *no* results (even if coverage defaults disagree). Seeing everything is almost never what wants to see if "--cov=MORF" was specified. Do you agree and see a way to implement this within pytest-cov? Another minor note: when using --cov-report=html no information is printed at all. Like the junitxml or figleaf plugin i suggest to print the location of the generated html directory or directly the index.html file. best & thanks for your nice work! holger > This is where source option comes in. With source option specify the > roots of trees then limited to those roots since measurement itself is > not done outside of those trees. > > :) > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- From memedough at gmail.com Tue Aug 3 14:43:30 2010 From: memedough at gmail.com (meme dough) Date: Tue, 3 Aug 2010 22:43:30 +1000 Subject: [py-dev] pytest-cov In-Reply-To: <20100803121700.GR1914@trillke.net> References: <20100802140543.GP1914@trillke.net> <20100803081428.GQ1914@trillke.net> <20100803121700.GR1914@trillke.net> Message-ID: Hi, > If i understand correctly you are describing the situation as is. > Sorry, i didn't look at the content of cov_core_init.py which would > have revealed the mechanics that you describe. Yes, that's how it is now. Not expecting you to dig into the code - just easier for me to paste it in rather than type it again :) > I guess somehow "pip" should be better able to uninstall. ?But ok, > cov_core_init.py is hopefully not causing trouble so it's not so bad. python setup.py install (have to manually remove) easy_install (have to manually remove) pip (if distutils / distutils2 / setuptools / distribute / pip could install and uninstall a pth file that would be great but I haven't seen way yet) >> --junitxml=path >> --ignore=path >> --genscript=path > > these are all file system paths. That is what it will be, but I can look at clarify it now. > This remains then the only issue i guess: rather give information > that the specified "--cov=XYZ" restriction was used and turn up > *no* results (even if coverage defaults disagree). ?Seeing everything > is almost never what wants to see if "--cov=MORF" was specified. > Do you agree and see a way to implement this within pytest-cov? That's interesting, I hadn't thought of it. Should be able to do it though I will look. > Another minor note: when using --cov-report=html no information > is printed at all. ?Like the junitxml or figleaf plugin i suggest > to print the location of the generated html directory or directly > the index.html file. That output could be useful and won't hurt if under continuous integration. I will look to see if possible to get the file path easily (could always get from config file or default but if can get from coverage better). :) From holger at merlinux.eu Tue Aug 3 15:01:27 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 Aug 2010 15:01:27 +0200 Subject: [py-dev] pytest-cov In-Reply-To: References: <20100802140543.GP1914@trillke.net> <20100803081428.GQ1914@trillke.net> <20100803121700.GR1914@trillke.net> Message-ID: <20100803130127.GS1914@trillke.net> On Tue, Aug 03, 2010 at 22:43 +1000, meme dough wrote: > Hi, > > > If i understand correctly you are describing the situation as is. > > Sorry, i didn't look at the content of cov_core_init.py which would > > have revealed the mechanics that you describe. > > Yes, that's how it is now. Not expecting you to dig into the code - > just easier for me to paste it in rather than type it again :) > > > I guess somehow "pip" should be better able to uninstall. ?But ok, > > cov_core_init.py is hopefully not causing trouble so it's not so bad. > > python setup.py install (have to manually remove) > easy_install (have to manually remove) > pip (if distutils / distutils2 / setuptools / distribute / pip could > install and uninstall a pth file that would be great but I haven't > seen way yet) Maybe you can change the IMPORTANT note to mention that the "cov-core" package is installing this init_cov_core.pth file so that people do not unneccessarily suspect the innocent pytest-cov plugin :) > >> --junitxml=path > >> --ignore=path > >> --genscript=path > > > > these are all file system paths. > > That is what it will be, but I can look at clarify it now. great. maybe just adapt the help string to "measure coverage for fs/import path (multi-allowed)" or so. > > This remains then the only issue i guess: rather give information > > that the specified "--cov=XYZ" restriction was used and turn up > > *no* results (even if coverage defaults disagree). ?Seeing everything > > is almost never what wants to see if "--cov=MORF" was specified. > > Do you agree and see a way to implement this within pytest-cov? > > That's interesting, I hadn't thought of it. Should be able to do it > though I will look. > > > Another minor note: when using --cov-report=html no information > > is printed at all. ?Like the junitxml or figleaf plugin i suggest > > to print the location of the generated html directory or directly > > the index.html file. > > That output could be useful and won't hurt if under continuous > integration. I will look to see if possible to get the file path > easily (could always get from config file or default but if can get > from coverage better). I think there even is some standard coverage reporting format that e.g. Hudson can read. Or it can at least grab the generated HTML. Never looked into it but would probably would be cool to have such reports available with CI runs :) best, holger From holger at merlinux.eu Wed Aug 4 12:39:08 2010 From: holger at merlinux.eu (holger krekel) Date: Wed, 4 Aug 2010 12:39:08 +0200 Subject: [py-dev] pytest-cov measuring multiple subprocesses? (+notes) Message-ID: <20100804103908.GT1914@trillke.net> Hi Meme, i am using pytest-cov for a project where I start multiple subprocesses. I am wondering how to best do this, here is the scenario: def test_functional(): popen1 = subprocess.Popen(...) # new python process popen2 = subprocess.Popen(...) # new python process popen1.terminate() popen2.wait() and now i want pytest-cov to show me the coverage. What do i need to do? Should it "just" work with this: py.test --cov=mypkg ? An unrelated note: the py.test distributed coverage-reporting feature ("DistMaster" etc.) is living with cov_core.py IISIC. Is this actually re-usable from nosetests? If not it would probably make more sense to have this code in pytest-cov for it to be easier to modify and maintain, no? another unrelated note: i like plugins to not cause import overhead if their functionality is not actually used. What about importing cov_core only when coverage is actually active? :) best, holger From memedough at gmail.com Thu Aug 5 12:51:42 2010 From: memedough at gmail.com (meme dough) Date: Thu, 5 Aug 2010 20:51:42 +1000 Subject: [py-dev] pytest-cov measuring multiple subprocesses? (+notes) In-Reply-To: <20100804103908.GT1914@trillke.net> References: <20100804103908.GT1914@trillke.net> Message-ID: Hi, > i am using pytest-cov for a project where I start multiple subprocesses. > I am wondering how to best do this, here is the scenario: > > ? ?def test_functional(): > ? ? ? ?popen1 = subprocess.Popen(...) # new python process > ? ? ? ?popen2 = subprocess.Popen(...) # new python process > ? ? ? ?popen1.terminate() > ? ? ? ?popen2.wait() > > > and now i want pytest-cov to show me the coverage. > What do i need to do? ?Should it "just" work with this: > > ? ?py.test --cov=mypkg > > ? Yes, it should just work provided that the subprocesses are python (as you state above) that have pytest-cov / cov-core installed. Only other thing is that it site initialisation as normal and env inherited as normal. Those sub processes detect env var and start coverage. Sub processes finish and write coverage file (specific per process), then master py.test finish and combine and report. > An unrelated note: the py.test distributed coverage-reporting feature > ("DistMaster" etc.) is living with cov_core.py IISIC. Is this actually re-usable > from nosetests? ?If not it would probably make more sense to have this code in > pytest-cov for it to be easier to modify and maintain, no? I liked having all the cov controllers together, but it is possible to move. > another unrelated note: i like plugins to not cause import overhead if > their functionality is not actually used. ?What about importing cov_core > only when coverage is actually active? > :) Yes this is good way. :) From jaraco at jaraco.com Thu Aug 5 17:05:49 2010 From: jaraco at jaraco.com (Jason R. Coombs) Date: Thu, 5 Aug 2010 08:05:49 -0700 Subject: [py-dev] feedback on py.test discovery/setup/process documentation Message-ID: <12C7AB425F0DD546B6049311F827C74E08A5FCE427@VA3DIAXVS141.RED001.local> I was espousing the joys of py.test at a recent #DCPython meetup. I found, however, that I'm not the only person frustrated by the lack of documentation on the basic process py.test uses to discover, setup, and conduct tests. There's mention here and there of setup/teardown methods, and their names are fairly intuitive, but I can't find that they're documented anywhere. It would be nice if there was a half-page document on the process py.test uses to conduct a single test. something like: # globally Run pytest_configure For each test module matching test_*.py: Run setup_module(module object) For each class in module.Test*: skip if it's not a valid test class (i.e. has an __init__ method) Run setup_class(class_object) [p.s. why doesn't setup_class have to be a classmethod?] For each method: Run setup_method(instance, method) Run test Run teardown_method Run teardown_class . I've discovered all of this through trial and error (and tracking down examples here and there). For example, I couldn't find setup_method documented anywhere, even though I knew what I wanted (a setup for each test to be run). Also, it would be nice to have some clarification on which classes will be discovered. I was recently bitten by an interesting feature. If a test class has an __init__ method, it will not be run as a test. For one test, I was using the __init__ method to define setup for each instance of the test. It turns out that by using setup_method instead of __init__, it did precisely what I wanted, but I went several weeks thinking my tests were running, when they were being silently skipped. I realize that all of this seems a little nitpicky. Truly, I love py.test, but I have to agree with my colleagues that it's a little difficult getting to know the basics. It doesn't help matters that because of the name, it's difficult to search the web for py.test resources. If you have any suggestions on how I might have learned these details other than through trial-and-error, please point me to them. Regards, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6448 bytes Desc: not available URL: From holger at merlinux.eu Thu Aug 5 17:28:55 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 5 Aug 2010 17:28:55 +0200 Subject: [py-dev] feedback on py.test discovery/setup/process documentation In-Reply-To: <12C7AB425F0DD546B6049311F827C74E08A5FCE427@VA3DIAXVS141.RED001.local> References: <12C7AB425F0DD546B6049311F827C74E08A5FCE427@VA3DIAXVS141.RED001.local> Message-ID: <20100805152854.GU1914@trillke.net> Hi Jason, thanks for the feedback, i appreciate it. I am aware that documentation needs some love - i actually plan to port it to sphinx (which has builtin search apart from other nicities) and add an example section. That being said, under http://codespeak.net/py/dist/test/features.html there is a link to "xUnit style setup": http://codespeak.net/py/dist/test/xunit_setup.html which may help a bit regarding the hooks. It doesn't mention pytest_configure which is mentioned here: http://codespeak.net/py/dist/test/customize.html ... so i guess it's true that things should be organised more according to usage patterns / needs rather than the structure of the implementation. Oh, and feel free to write blog posts about things you found so we can link them :) best, holger On Thu, Aug 05, 2010 at 08:05 -0700, Jason R. Coombs wrote: > I was espousing the joys of py.test at a recent #DCPython meetup. I found, > however, that I'm not the only person frustrated by the lack of > documentation on the basic process py.test uses to discover, setup, and > conduct tests. > > > > There's mention here and there of setup/teardown methods, and their names > are fairly intuitive, but I can't find that they're documented anywhere. It > would be nice if there was a half-page document on the process py.test uses > to conduct a single test. something like: > > > > # globally > > Run pytest_configure > > For each test module matching test_*.py: > > Run setup_module(module object) > > For each class in module.Test*: > > skip if it's not a valid test class (i.e. > has an __init__ method) > > Run setup_class(class_object) [p.s. why > doesn't setup_class have to be a classmethod?] > > For each method: > > Run setup_method(instance, > method) > > Run test > > Run teardown_method > > Run teardown_class > > . > > > > I've discovered all of this through trial and error (and tracking down > examples here and there). For example, I couldn't find setup_method > documented anywhere, even though I knew what I wanted (a setup for each test > to be run). > > > > Also, it would be nice to have some clarification on which classes will be > discovered. I was recently bitten by an interesting feature. If a test class > has an __init__ method, it will not be run as a test. For one test, I was > using the __init__ method to define setup for each instance of the test. It > turns out that by using setup_method instead of __init__, it did precisely > what I wanted, but I went several weeks thinking my tests were running, when > they were being silently skipped. > > > > I realize that all of this seems a little nitpicky. Truly, I love py.test, > but I have to agree with my colleagues that it's a little difficult getting > to know the basics. It doesn't help matters that because of the name, it's > difficult to search the web for py.test resources. > > > > If you have any suggestions on how I might have learned these details other > than through trial-and-error, please point me to them. > > > > Regards, > > > > Jason > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -- From jaraco at jaraco.com Thu Aug 5 22:46:24 2010 From: jaraco at jaraco.com (Jason R. Coombs) Date: Thu, 5 Aug 2010 13:46:24 -0700 Subject: [py-dev] feedback on py.test discovery/setup/process documentation In-Reply-To: <20100805152854.GU1914@trillke.net> References: <12C7AB425F0DD546B6049311F827C74E08A5FCE427@VA3DIAXVS141.RED001.local> <20100805152854.GU1914@trillke.net> Message-ID: <12C7AB425F0DD546B6049311F827C74E08A60DC9D8@VA3DIAXVS141.RED001.local> That's a big help. I had bypassed the xUnit docs because the name sounded like a feature in which I wasn't interested (just because of the name). Silly me. Thanks for the reference. > -----Original Message----- > From: holger krekel [mailto:holger at merlinux.eu] > Sent: Thursday, 05 August, 2010 11:29 > To: Jason R. Coombs > Cc: py-dev at codespeak.net > Subject: Re: [py-dev] feedback on py.test discovery/setup/process > documentation > > Hi Jason, > > thanks for the feedback, i appreciate it. I am aware that documentation > needs some love - i actually plan to port it to sphinx (which has builtin search > apart from other nicities) and add an example section. > > That being said, under > > http://codespeak.net/py/dist/test/features.html > > there is a link to "xUnit style setup": > > http://codespeak.net/py/dist/test/xunit_setup.html > > which may help a bit regarding the hooks. It doesn't mention > pytest_configure which is mentioned here: > > http://codespeak.net/py/dist/test/customize.html > > ... so i guess it's true that things should be organised more according to usage > patterns / needs rather than the structure of the implementation. Oh, and > feel free to write blog posts about things you found so we can link them :) > > best, > holger > > On Thu, Aug 05, 2010 at 08:05 -0700, Jason R. Coombs wrote: > > I was espousing the joys of py.test at a recent #DCPython meetup. I > > found, however, that I'm not the only person frustrated by the lack of > > documentation on the basic process py.test uses to discover, setup, > > and conduct tests. > > > > > > > > There's mention here and there of setup/teardown methods, and their > > names are fairly intuitive, but I can't find that they're documented > > anywhere. It would be nice if there was a half-page document on the > > process py.test uses to conduct a single test. something like: > > > > > > > > # globally > > > > Run pytest_configure > > > > For each test module matching test_*.py: > > > > Run setup_module(module object) > > > > For each class in module.Test*: > > > > skip if it's not a valid test class (i.e. > > has an __init__ method) > > > > Run setup_class(class_object) [p.s. > > why doesn't setup_class have to be a classmethod?] > > > > For each method: > > > > Run > > setup_method(instance, > > method) > > > > Run test > > > > Run teardown_method > > > > Run teardown_class > > > > . > > > > > > > > I've discovered all of this through trial and error (and tracking down > > examples here and there). For example, I couldn't find setup_method > > documented anywhere, even though I knew what I wanted (a setup for > > each test to be run). > > > > > > > > Also, it would be nice to have some clarification on which classes > > will be discovered. I was recently bitten by an interesting feature. > > If a test class has an __init__ method, it will not be run as a test. > > For one test, I was using the __init__ method to define setup for each > > instance of the test. It turns out that by using setup_method instead > > of __init__, it did precisely what I wanted, but I went several weeks > > thinking my tests were running, when they were being silently skipped. > > > > > > > > I realize that all of this seems a little nitpicky. Truly, I love > > py.test, but I have to agree with my colleagues that it's a little > > difficult getting to know the basics. It doesn't help matters that > > because of the name, it's difficult to search the web for py.test resources. > > > > > > > > If you have any suggestions on how I might have learned these details > > other than through trial-and-error, please point me to them. > > > > > > > > Regards, > > > > > > > > Jason > > > > > > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > > > -- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6448 bytes Desc: not available URL: From sridharr at activestate.com Thu Aug 5 22:46:14 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 05 Aug 2010 13:46:14 -0700 Subject: [py-dev] py.test api - Running tests in a module object? Message-ID: <4C5B2316.7050909@activestate.com> I am trying to run py.test on a given module object dynamically. mod = __import__('company.foo.somemod') py.test.run(mod) A quick glance in the py.test source doesn't seem like a trivial thing. IPython introspection did not give me any clue either. Is there a way to do this at all? I did try py.test.cmdline.main(['...']), but that accepts only file path, not the Python module object itself. -srid From holger at merlinux.eu Thu Aug 5 23:06:52 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 5 Aug 2010 23:06:52 +0200 Subject: [py-dev] py.test api - Running tests in a module object? In-Reply-To: <4C5B2316.7050909@activestate.com> References: <4C5B2316.7050909@activestate.com> Message-ID: <20100805210652.GY1914@trillke.net> Hi Sridhar, On Thu, Aug 05, 2010 at 13:46 -0700, Sridhar Ratnakumar wrote: > I am trying to run py.test on a given module object dynamically. > > mod = __import__('company.foo.somemod') > py.test.run(mod) > > A quick glance in the py.test source doesn't seem like a trivial thing. > IPython introspection did not give me any clue either. > > Is there a way to do this at all? I did try > py.test.cmdline.main(['...']), but that accepts only file path, not the > Python module object itself. there currently is no direct support for running tests in python modules. py.test basically always starts from the file system. Does it help you to try to fish the file from somemod.__file__ and pass this to cmdline.main()? You only expect it to collect tests of a single module, right? I guess we could add some more direct support for this if you continue to have the need (please create an issue if so). cheers, holger From sridharr at activestate.com Thu Aug 5 23:24:07 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 05 Aug 2010 14:24:07 -0700 Subject: [py-dev] py.test api - Running tests in a module object? In-Reply-To: <20100805210652.GY1914@trillke.net> References: <4C5B2316.7050909@activestate.com> <20100805210652.GY1914@trillke.net> Message-ID: <4C5B2BF7.8070107@activestate.com> On 8/5/2010 2:06 PM, holger krekel wrote: > On Thu, Aug 05, 2010 at 13:46 -0700, Sridhar Ratnakumar wrote: >> > I am trying to run py.test on a given module object dynamically. >> > >> > mod = __import__('company.foo.somemod') >> > py.test.run(mod) >> > >> > A quick glance in the py.test source doesn't seem like a trivial thing. >> > IPython introspection did not give me any clue either. >> > >> > Is there a way to do this at all? I did try >> > py.test.cmdline.main(['...']), but that accepts only file path, not the >> > Python module object itself. > there currently is no direct support for running tests in python modules. > py.test basically always starts from the file system. Does it help you > to try to fish the file from somemod.__file__ and pass this to cmdline.main()? > You only expect it to collect tests of a single module, right? > I guess we could add some more direct support for this if > you continue to have the need (please create an issue if so). Hmm, __file__ is a workaround, but it seems to work fine for my use case. I was just wondering if an API was exposed. I am happy with __file__ for now. -srid From holger at merlinux.eu Thu Aug 5 23:28:23 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 5 Aug 2010 23:28:23 +0200 Subject: [py-dev] py.test api - Running tests in a module object? In-Reply-To: <4C5B2BF7.8070107@activestate.com> References: <4C5B2316.7050909@activestate.com> <20100805210652.GY1914@trillke.net> <4C5B2BF7.8070107@activestate.com> Message-ID: <20100805212823.GB1914@trillke.net> On Thu, Aug 05, 2010 at 14:24 -0700, Sridhar Ratnakumar wrote: > On 8/5/2010 2:06 PM, holger krekel wrote: >> On Thu, Aug 05, 2010 at 13:46 -0700, Sridhar Ratnakumar wrote: >>> > I am trying to run py.test on a given module object dynamically. >>> > >>> > mod = __import__('company.foo.somemod') >>> > py.test.run(mod) >>> > >>> > A quick glance in the py.test source doesn't seem like a trivial thing. >>> > IPython introspection did not give me any clue either. >>> > >>> > Is there a way to do this at all? I did try >>> > py.test.cmdline.main(['...']), but that accepts only file path, not the >>> > Python module object itself. >> there currently is no direct support for running tests in python modules. >> py.test basically always starts from the file system. Does it help you >> to try to fish the file from somemod.__file__ and pass this to cmdline.main()? >> You only expect it to collect tests of a single module, right? >> I guess we could add some more direct support for this if >> you continue to have the need (please create an issue if so). > > Hmm, __file__ is a workaround, but it seems to work fine for my use > case. I was just wondering if an API was exposed. > > I am happy with __file__ for now. What do you think about automating this lookup and allowing py.test.cmdline.main([module]) maybe that's enough? best, holger From sridharr at activestate.com Thu Aug 5 23:39:42 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 5 Aug 2010 14:39:42 -0700 Subject: [py-dev] py.test api - Running tests in a module object? In-Reply-To: <20100805212823.GB1914@trillke.net> References: <4C5B2316.7050909@activestate.com> <20100805210652.GY1914@trillke.net> <4C5B2BF7.8070107@activestate.com> <20100805212823.GB1914@trillke.net> Message-ID: <10C7D726-EFFB-42F4-902C-D023A20FE498@activestate.com> On 2010-08-05, at 2:28 PM, holger krekel wrote: > On Thu, Aug 05, 2010 at 14:24 -0700, Sridhar Ratnakumar wrote: >> On 8/5/2010 2:06 PM, holger krekel wrote: >>> On Thu, Aug 05, 2010 at 13:46 -0700, Sridhar Ratnakumar wrote: >>>>> I am trying to run py.test on a given module object dynamically. >>>>> >>>>> mod = __import__('company.foo.somemod') >>>>> py.test.run(mod) >>>>> >>>>> A quick glance in the py.test source doesn't seem like a trivial thing. >>>>> IPython introspection did not give me any clue either. >>>>> >>>>> Is there a way to do this at all? I did try >>>>> py.test.cmdline.main(['...']), but that accepts only file path, not the >>>>> Python module object itself. >>> there currently is no direct support for running tests in python modules. >>> py.test basically always starts from the file system. Does it help you >>> to try to fish the file from somemod.__file__ and pass this to cmdline.main()? >>> You only expect it to collect tests of a single module, right? >>> I guess we could add some more direct support for this if >>> you continue to have the need (please create an issue if so). >> >> Hmm, __file__ is a workaround, but it seems to work fine for my use >> case. I was just wondering if an API was exposed. >> >> I am happy with __file__ for now. > > What do you think about automating this lookup and allowing > > py.test.cmdline.main([module]) > > maybe that's enough? That might have helped, but then - just now, I rewrote some routines in my code to simply rely on .py file paths instead of modules. This simplified the code a bit. So now I have come to prefer file paths. :-) If I ever have the need for running tests on module/class/collection-of-funcs objects, I will be sure to open a feature request. Thanks. -srid From issues-noreply at bitbucket.org Fri Aug 6 01:46:12 2010 From: issues-noreply at bitbucket.org (issues-noreply at bitbucket.org) Date: Thu, 05 Aug 2010 23:46:12 -0000 Subject: [py-dev] New issue 114 in py-trunk: cmdline(..) uses wrong program name Message-ID: --- you can reply above this line --- New issue 114: cmdline(..) uses wrong program name http://bitbucket.org/hpk42/py-trunk/issue/114/cmdline-uses-wrong-program-name Sridhar Ratnakumar / srid on Fri, 6 Aug 2010 01:46:12 +0200: Description: I am invoking py.test via py.test.cmdline(). When there is an error in command line, the following is printed: {{{ usage: boa [options] [file_or_dir] [file_or_dir] [...] boa: error: no such option: --junitlxml }}} "boa" is the name of *my* program, which py.test.cmdline() is stealing. Responsible: hpk42 -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From flub at devork.be Sat Aug 7 11:52:24 2010 From: flub at devork.be (Floris Bruynooghe) Date: Sat, 7 Aug 2010 10:52:24 +0100 Subject: [py-dev] Generic funcarg hook Message-ID: <20100807095224.GA14232@laurie.devork.be> Hi I was talking to ronny on IRC about wanting a generic funcarg hook and he asked me to post my use case here as you are still thinking about the API for it. Since there is no generic funcarg hook now this is how I currently create the tests: def pytest_funcarg__tdata(request): fname = os.path.join(DATADIR, requst.function.__name__+'.ext') with file(fname) as fd: obj = fd.read() return obj def test_spam(tdata): assert func_under_test(tdata) == 'foo' def test_eggs(tdata): assert func_under_test(tdata) == 'bar' Before I started writing it this way I was looking for a generic funcarg hook since I think this would have been clearer if I could have used "spam" and "eggs" as funcargs and still create the object for them from one function rather then having to spell it out for each one. So I'm hoping that in a future version you might add an API to do allow something like this. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From memedough at gmail.com Sun Aug 8 10:27:53 2010 From: memedough at gmail.com (meme dough) Date: Sun, 8 Aug 2010 18:27:53 +1000 Subject: [py-dev] pytest-cov Message-ID: Hi, The --cov option is now filesystem paths as that should be how it will work with the upcoming source option from coverage. Help text updated. Warning when nothing to report on. Reports on generation of html, xml and annotated source. Minimised imports. Haven't moved DistMaster / DistSlave yet as found easier for minimising imports if not in pytest-cov, may make a separate module there, will consider further. :) From holger at merlinux.eu Tue Aug 10 10:50:00 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 10 Aug 2010 10:50:00 +0200 Subject: [py-dev] pytest-cov In-Reply-To: References: Message-ID: <20100810085000.GD1914@trillke.net> Hi Meme, On Sun, Aug 08, 2010 at 18:27 +1000, meme dough wrote: > Hi, > > The --cov option is now filesystem paths as that should be how it will > work with the upcoming source option from coverage. Help text > updated. FYI i am fine (and interested, i guess) in both versions: specifying a file system path as well as a python import path. If in doubt filesystem is probably more consistent with py.test's way of looking at tests (almost all related options specify fs paths) so i guess this is ok for now. will try your latest release latest when back from holiday which i am going for today for 2-3 weeks :) best, holger > Warning when nothing to report on. > > Reports on generation of html, xml and annotated source. > > Minimised imports. > > Haven't moved DistMaster / DistSlave yet as found easier for > minimising imports if not in pytest-cov, may make a separate module > there, will consider further. > > :) > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- From holger at merlinux.eu Tue Aug 10 11:01:09 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 10 Aug 2010 11:01:09 +0200 Subject: [py-dev] Generic funcarg hook In-Reply-To: <20100807095224.GA14232@laurie.devork.be> References: <20100807095224.GA14232@laurie.devork.be> Message-ID: <20100810090109.GF1914@trillke.net> Hi Floris, to be sure i understand your use-case: could you invent an API/example that you'd like to see working? FYI, i am considering introducing a slightly higher level mechanism on top of funcargs, dealing with (parametrizable) resources. Something like: @py.test.mark.resourcefactory(scope="session", argnames=['foo', 'bar']) def myfactory(request): if request.argname == "foo": ... elif request.argname == "bar": ... The request object would be what pytest_funcarg__NAME get but this would allow free naming and have a single factory become responsible for multiple funcargs and allow to declare scope reather than having to invoke request.cached_setup(). Instead of the latter, one could write "request.addfinalizer()" which would default to the declared scope. Happy about any thoughts on this but mainly i mention this consideration here in case it would fit with your use case. best, holger On Sat, Aug 07, 2010 at 10:52 +0100, Floris Bruynooghe wrote: > Hi > > I was talking to ronny on IRC about wanting a generic funcarg hook and > he asked me to post my use case here as you are still thinking about > the API for it. Since there is no generic funcarg hook now this is > how I currently create the tests: > > > def pytest_funcarg__tdata(request): > fname = os.path.join(DATADIR, requst.function.__name__+'.ext') > with file(fname) as fd: > obj = fd.read() > return obj > > def test_spam(tdata): > assert func_under_test(tdata) == 'foo' > > def test_eggs(tdata): > assert func_under_test(tdata) == 'bar' > > > Before I started writing it this way I was looking for a generic > funcarg hook since I think this would have been clearer if I could > have used "spam" and "eggs" as funcargs and still create the object > for them from one function rather then having to spell it out for each > one. So I'm hoping that in a future version you might add an API to > do allow something like this. > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- From holger at merlinux.eu Tue Aug 10 12:10:13 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 10 Aug 2010 12:10:13 +0200 Subject: [py-dev] parametrized resources (Re: Generic funcarg hook) In-Reply-To: <20100810090109.GF1914@trillke.net> References: <20100807095224.GA14232@laurie.devork.be> <20100810090109.GF1914@trillke.net> Message-ID: <20100810101013.GG1914@trillke.net> Hi Floris, all, On Tue, Aug 10, 2010 at 11:01 +0200, holger krekel wrote: > FYI, i am considering introducing a slightly higher level > mechanism on top of funcargs, dealing with (parametrizable) > resources. Something like: > > @py.test.mark.resourcefactory(scope="session", argnames=['foo', 'bar']) > def myfactory(request): > if request.argname == "foo": > ... > elif request.argname == "bar": > ... > > The request object would be what pytest_funcarg__NAME get but > this would allow free naming and have a single factory become > responsible for multiple funcargs and allow to declare scope > reather than having to invoke request.cached_setup(). Instead > of the latter, one could write "request.addfinalizer()" which > would default to the declared scope. just to be clear, the test function would look like: def test_function(foo): ... foo is initialized per-session, created per the factory above def test_function2(foo, bar): ... foo and bare are created by two invocations to the factory above Moreover, regarding the "parametrized resources" aspect, there would be a new decorator: @py.test.mark.parametrize(foo=iterable) def test_function(foo): ... invoked for multiple foo instances The idea is to be able to avoid pytest_generate_tests functions in a number of cases. Both the current funcarg-mechanism and the generate-tests mechanism would continue to exist but be considered "lower level", i.e. upcoming examples on the web page would use the new more declarative approach, requiring less API knowledge and naming freedom to use them. any comments and particularly real use cases very welcome! best, holger > Happy about any thoughts on this but mainly i mention this > consideration here in case it would fit with your use case. > > best, > holger > > > On Sat, Aug 07, 2010 at 10:52 +0100, Floris Bruynooghe wrote: > > Hi > > > > I was talking to ronny on IRC about wanting a generic funcarg hook and > > he asked me to post my use case here as you are still thinking about > > the API for it. Since there is no generic funcarg hook now this is > > how I currently create the tests: > > > > > > def pytest_funcarg__tdata(request): > > fname = os.path.join(DATADIR, requst.function.__name__+'.ext') > > with file(fname) as fd: > > obj = fd.read() > > return obj > > > > def test_spam(tdata): > > assert func_under_test(tdata) == 'foo' > > > > def test_eggs(tdata): > > assert func_under_test(tdata) == 'bar' > > > > > > Before I started writing it this way I was looking for a generic > > funcarg hook since I think this would have been clearer if I could > > have used "spam" and "eggs" as funcargs and still create the object > > for them from one function rather then having to spell it out for each > > one. So I'm hoping that in a future version you might add an API to > > do allow something like this. > > > > -- > > Debian GNU/Linux -- The Power of Freedom > > www.debian.org | www.gnu.org | www.kernel.org > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > > > > -- > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- From holger at merlinux.eu Wed Aug 11 09:28:41 2010 From: holger at merlinux.eu (holger krekel) Date: Wed, 11 Aug 2010 09:28:41 +0200 Subject: [py-dev] survey results unittest/py.test/nose/... Message-ID: <20100811072841.GJ1914@trillke.net> Hi all, came across this activestate link which recently surveyed usage of test automation frameworks: http://www.activestate.com/blog/2010/07/survey-says-selenium-nose-most-popular-test-framework I am pleasantly surprised to find py.test come ahead of nose and don't quite get the title and the summary which seem to indicate otherwise. Probably I am missing something? best, holger From flub at devork.be Wed Aug 11 19:58:22 2010 From: flub at devork.be (Floris Bruynooghe) Date: Wed, 11 Aug 2010 18:58:22 +0100 Subject: [py-dev] Generic funcarg hook In-Reply-To: <20100810090109.GF1914@trillke.net> References: <20100807095224.GA14232@laurie.devork.be> <20100810090109.GF1914@trillke.net> Message-ID: <20100811175822.GA28339@laurie.devork.be> Hello Holger On Tue, Aug 10, 2010 at 11:01:09AM +0200, holger krekel wrote: > to be sure i understand your use-case: could you invent > an API/example that you'd like to see working? def pytest_funcarg_factory(request): if request.funcargname == 'foo': return object() elif request.funcargname == 'bar': return object() else: raise KeyError() And when it raises a KeyError (or does not exist) it would fall back to the normal pytest_funcarg__name functions. So using a factory like this would still be the exception rather then the rule, most funcargs would come from pytest_funcarg__name functions. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From flub at devork.be Wed Aug 11 20:28:43 2010 From: flub at devork.be (Floris Bruynooghe) Date: Wed, 11 Aug 2010 19:28:43 +0100 Subject: [py-dev] parametrized resources (Re: Generic funcarg hook) In-Reply-To: <20100810101013.GG1914@trillke.net> References: <20100807095224.GA14232@laurie.devork.be> <20100810090109.GF1914@trillke.net> <20100810101013.GG1914@trillke.net> Message-ID: <20100811182843.GB28339@laurie.devork.be> On Tue, Aug 10, 2010 at 12:10:13PM +0200, holger krekel wrote: > On Tue, Aug 10, 2010 at 11:01 +0200, holger krekel wrote: > > FYI, i am considering introducing a slightly higher level > > mechanism on top of funcargs, dealing with (parametrizable) > > resources. Something like: > > > > @py.test.mark.resourcefactory(scope="session", argnames=['foo', 'bar']) > > def myfactory(request): > > if request.argname == "foo": > > ... > > elif request.argname == "bar": > > ... [...] > Moreover, regarding the "parametrized resources" aspect, > there would be a new decorator: > > @py.test.mark.parametrize(foo=iterable) > def test_function(foo): > ... invoked for multiple foo instances Not sure how much I like these examples. But the use of decorators could make things clearer maybe, I think something like this might make things easier to write: @py.test.cached(scope='session') # or py.test.mark.cached def pytest_funcarg__foo(request): pass Here the actual function would only be called once for the session. @py.test.parametrize(iterable) def pytest_funcarg__foo(request, args, from, iterable): pass In this case the function would be called as many times as the iterable has items. Each item in the iterable would be the arguments to pass to the function. The iterable might have to work both with ['arg1', 'arg2', ...] as well as [('spam1', 'eggs1'), ('spam2', 'eggs2'), ...] to be easy to use. And of course nesting these decorators would be useful too. Here's how I'd use this in real code I have: @py.test.cached(scope='session') @py.test.parametrize(['v1', 'v2c', 'v3']) def pytest_funcarg__snmpcfg(request, version): if version in ('v1', 'v2c'): cfg = check_snmp_v12(version) elif version == 'v3': cfg = check_snmp_v3() if not cfg: py.test.skip('No SNMP%s agent available' % version) return SnmpWrapper(cfg) I expect this function to be called 3 times during the whole session and any test function asking for "snmpcfg" will get called 3 times (some of those 3 might skip instead in this case). This should probably work independed of the order of the decorators (the order shown reads a little funny I guess). Anyway, just my toughts after reading your description Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From fuzzyman at voidspace.org.uk Fri Aug 13 16:28:54 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 13 Aug 2010 15:28:54 +0100 Subject: [py-dev] parametrized resources (Re: Generic funcarg hook) In-Reply-To: <20100810101013.GG1914@trillke.net> References: <20100807095224.GA14232@laurie.devork.be> <20100810090109.GF1914@trillke.net> <20100810101013.GG1914@trillke.net> Message-ID: <4C6556A6.5040708@voidspace.org.uk> On 10/08/2010 11:10, holger krekel wrote: > Hi Floris, all, > > On Tue, Aug 10, 2010 at 11:01 +0200, holger krekel wrote: > >> FYI, i am considering introducing a slightly higher level >> mechanism on top of funcargs, dealing with (parametrizable) >> resources. Something like: >> >> @py.test.mark.resourcefactory(scope="session", argnames=['foo', 'bar']) >> def myfactory(request): >> if request.argname == "foo": >> ... >> elif request.argname == "bar": >> ... >> >> The request object would be what pytest_funcarg__NAME get but >> this would allow free naming and have a single factory become >> responsible for multiple funcargs and allow to declare scope >> reather than having to invoke request.cached_setup(). Instead >> of the latter, one could write "request.addfinalizer()" which >> would default to the declared scope. >> > just to be clear, the test function would look like: > > def test_function(foo): > ... foo is initialized per-session, created per the factory above > > def test_function2(foo, bar): > ... foo and bare are created by two invocations to the factory above > > Moreover, regarding the "parametrized resources" aspect, > there would be a new decorator: > > @py.test.mark.parametrize(foo=iterable) > def test_function(foo): > ... invoked for multiple foo instances > > The idea is to be able to avoid pytest_generate_tests functions > in a number of cases. > > Hey all, For unittest(2) there is an implementation of parametrized tests that already exists as a plugin. http://hg.python.org/unittest2/file/tip/unittest2/plugins/moduleloading.py The decorator form is basically identical to the suggested form above: from unittest2.plugins.moduleloading import params @params(iterable) def test_function(arg1, arg2, arg3): ... test_function will be expanded into one test per member in iterable at test load time. This is all experimental, i.e. we don't know yet if any of this will come into core unittest or live as a plugin forever. The plugin also supports test generators, but unlike test generators in nose these are expanded at test *load* time rather than test execution time. They are supported at the moment because a lot of the people really like the generator syntax for creating tests. from unittest2.plugins.moduleloading import testGenerator @ testGenerator def test_function(): for value in range(5): yield other_function, (value,) All the best, Michael > Both the current funcarg-mechanism and the generate-tests mechanism > would continue to exist but be considered "lower level", i.e. upcoming > examples on the web page would use the new more declarative approach, > requiring less API knowledge and naming freedom to use them. > > any comments and particularly real use cases very welcome! > > best, > holger > > >> Happy about any thoughts on this but mainly i mention this >> consideration here in case it would fit with your use case. >> >> best, >> holger >> >> >> On Sat, Aug 07, 2010 at 10:52 +0100, Floris Bruynooghe wrote: >> >>> Hi >>> >>> I was talking to ronny on IRC about wanting a generic funcarg hook and >>> he asked me to post my use case here as you are still thinking about >>> the API for it. Since there is no generic funcarg hook now this is >>> how I currently create the tests: >>> >>> >>> def pytest_funcarg__tdata(request): >>> fname = os.path.join(DATADIR, requst.function.__name__+'.ext') >>> with file(fname) as fd: >>> obj = fd.read() >>> return obj >>> >>> def test_spam(tdata): >>> assert func_under_test(tdata) == 'foo' >>> >>> def test_eggs(tdata): >>> assert func_under_test(tdata) == 'bar' >>> >>> >>> Before I started writing it this way I was looking for a generic >>> funcarg hook since I think this would have been clearer if I could >>> have used "spam" and "eggs" as funcargs and still create the object >>> for them from one function rather then having to spell it out for each >>> one. So I'm hoping that in a future version you might add an API to >>> do allow something like this. >>> >>> -- >>> Debian GNU/Linux -- The Power of Freedom >>> www.debian.org | www.gnu.org | www.kernel.org >>> _______________________________________________ >>> py-dev mailing list >>> py-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/py-dev >>> >>> >> -- >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev >> >> > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From issues-noreply at bitbucket.org Sun Aug 15 16:29:43 2010 From: issues-noreply at bitbucket.org (issues-noreply at bitbucket.org) Date: Sun, 15 Aug 2010 14:29:43 -0000 Subject: [py-dev] New issue 115 in py-trunk: Catch-all exceptions don't make an exception for GeneratorExit Message-ID: --- you can reply above this line --- New issue 115: Catch-all exceptions don't make an exception for GeneratorExit http://bitbucket.org/hpk42/py-trunk/issue/115/catch-all-exceptions-dont-make-an-exception-for Floris Bruynooghe / flub on Sun, 15 Aug 2010 16:29:43 +0200: Description: Hi pylib seems to often use {{{ try: ... except (KeyboardInterrupt, SystemExit): raise except: ... }}} However this currently misses GeneratorExit and might miss other exceptions which don't (and shouldn't) inherit from Exception in the future. However the hierarchy only got rearranged in python 2.5 and GeneratorExit was only present since 2.5. So maybe this can't be solved until 2.4 is no longer supported. Regards Floris Responsible: hpk42 -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From flub at devork.be Mon Aug 16 02:25:57 2010 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 16 Aug 2010 01:25:57 +0100 Subject: [py-dev] Advanced assert equal Message-ID: <20100816002557.GA12335@laurie.devork.be> Hi Ever since unittest grew it's .assertSequenceEqual() and .assertMultilineEqual() I've been jealous of it. So this weekend I've looked into the py.test code and made an attempt at getting this into my favourite testing tool. The attached patch makes compare equal a special case and checks if the two arguments to it are both a list, text or dict and tries to generate a nicer explanation text for them. The patch is more like a proof of concept then a final implementation, I may have done some very strange or silly things as I'm not familiar with the code. It would be great to get feedback, both on the general concept and the actual implementation (particularly note the way I had to hack _format_explanation() in assertion.py). Some of the rough edges I can think off right now: (i) no idea how comparisons and nested calls work together, (ii) no attempt is made to limit the output from difflib so the screen doesn't get flooded. There's probably many more. I hope this can be useful Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org -------------- next part -------------- A non-text attachment was scrubbed... Name: py.test.diff Type: text/x-diff Size: 6744 bytes Desc: not available URL: From holger at merlinux.eu Mon Aug 16 09:10:16 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 16 Aug 2010 09:10:16 +0200 Subject: [py-dev] Advanced assert equal In-Reply-To: <20100816002557.GA12335@laurie.devork.be> References: <20100816002557.GA12335@laurie.devork.be> Message-ID: <20100816071015.GR1914@trillke.net> Hey Floris, nice. i also thought about improving reporting for particular types of assert-expressions. Will take a look at your code after holiday and maybe Benjamin can also take a look or make a suggestion on how to best make assert expression-reporting customizable. cheers, holger On Mon, Aug 16, 2010 at 01:25 +0100, Floris Bruynooghe wrote: > Hi > > Ever since unittest grew it's .assertSequenceEqual() and > .assertMultilineEqual() I've been jealous of it. So this weekend I've > looked into the py.test code and made an attempt at getting this into > my favourite testing tool. > > The attached patch makes compare equal a special case and checks if > the two arguments to it are both a list, text or dict and tries to > generate a nicer explanation text for them. The patch is more like a > proof of concept then a final implementation, I may have done some > very strange or silly things as I'm not familiar with the code. It > would be great to get feedback, both on the general concept and the > actual implementation (particularly note the way I had to hack > _format_explanation() in assertion.py). > > Some of the rough edges I can think off right now: (i) no idea how > comparisons and nested calls work together, (ii) no attempt is made to > limit the output from difflib so the screen doesn't get flooded. > There's probably many more. > > I hope this can be useful > Floris > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > diff --git a/py/_code/_assertionnew.py b/py/_code/_assertionnew.py > --- a/py/_code/_assertionnew.py > +++ b/py/_code/_assertionnew.py > @@ -5,6 +5,8 @@ This should replace _assertionold.py eve > > import sys > import ast > +import difflib > +import pprint > > import py > from py._code.assertion import _format_explanation, BuiltinAssertionError > @@ -164,8 +166,6 @@ class DebugInterpreter(ast.NodeVisitor): > left_explanation, left_result = self.visit(left) > got_result = False > for op, next_op in zip(comp.ops, comp.comparators): > - if got_result and not result: > - break > next_explanation, next_result = self.visit(next_op) > op_symbol = operator_map[op.__class__] > explanation = "%s %s %s" % (left_explanation, op_symbol, > @@ -177,11 +177,56 @@ class DebugInterpreter(ast.NodeVisitor): > __exprinfo_right=next_result) > except Exception: > raise Failure(explanation) > - else: > - got_result = True > + if not result: > + break > left_explanation, left_result = next_explanation, next_result > + if op_symbol == "==": > + new_expl = self._explain_equal(left_result, next_result, > + left_explanation, next_explanation) > + if new_expl: > + explanation = new_expl > return explanation, result > > + def _explain_equal(self, left, right, left_repr, right_repr): > + """Make a specialised explanation for comapare equal""" > + if type(left) != type(right): > + return None > + explanation = [] > + if len(left_repr) > 30: > + left_repr = left_repr[:27] + '...' > + if len(right_repr) > 30: > + right_repr = right_repr[:27] + '...' > + explanation += ['%s == %s' % (left_repr, right_repr)] > + issquence = lambda x: isinstance(x, (list, tuple)) > + istext = lambda x: isinstance(x, basestring) > + isdict = lambda x: isinstance(x, dict) > + if issquence(left): > + for i in xrange(min(len(left), len(right))): > + if left[i] != right[i]: > + explanation += ['First differing item %s: %s != %s' % > + (i, left[i], right[i])] > + break > + if len(left) > len(right): > + explanation += ['Left contains more items, ' > + 'first extra item: %s' % left[len(right)]] > + elif len(left) < len(right): > + explanation += ['Right contains more items, ' > + 'first extra item: %s' % right[len(right)]] > + explanation += [line.strip('\n') for line in > + difflib.ndiff(pprint.pformat(left).splitlines(), > + pprint.pformat(right).splitlines())] > + elif istext(left): > + explanation += [line.strip('\n') for line in > + difflib.ndiff(left.splitlines(), > + right.splitlines())] > + elif isdict(left): > + explanation += [line.strip('\n') for line in > + difflib.ndiff(pprint.pformat(left).splitlines(), > + pprint.pformat(right).splitlines())] > + else: > + return None # No specialised knowledge > + return '\n=='.join(explanation) > + > def visit_BoolOp(self, boolop): > is_or = isinstance(boolop.op, ast.Or) > explanations = [] > diff --git a/py/_code/assertion.py b/py/_code/assertion.py > --- a/py/_code/assertion.py > +++ b/py/_code/assertion.py > @@ -10,7 +10,7 @@ def _format_explanation(explanation): > # escape newlines not followed by { and } > lines = [raw_lines[0]] > for l in raw_lines[1:]: > - if l.startswith('{') or l.startswith('}'): > + if l.startswith('{') or l.startswith('}') or l.startswith('=='): > lines.append(l) > else: > lines[-1] += '\\n' + l > @@ -28,11 +28,14 @@ def _format_explanation(explanation): > stackcnt[-1] += 1 > stackcnt.append(0) > result.append(' +' + ' '*(len(stack)-1) + s + line[1:]) > - else: > + elif line.startswith('}'): > assert line.startswith('}') > stack.pop() > stackcnt.pop() > result[stack[-1]] += line[1:] > + else: > + assert line.startswith('==') > + result.append(' ' + line.strip('==')) > assert len(stack) == 1 > return '\n'.join(result) > > diff --git a/testing/code/test_assertionnew.py b/testing/code/test_assertionnew.py > new file mode 100644 > --- /dev/null > +++ b/testing/code/test_assertionnew.py > @@ -0,0 +1,74 @@ > +import sys > + > +import py > +from py._code._assertionnew import interpret > + > + > +def getframe(): > + """Return the frame of the caller as a py.code.Frame object""" > + return py.code.Frame(sys._getframe(1)) > + > + > +def setup_module(mod): > + py.code.patch_builtins(assertion=True, compile=False) > + > + > +def teardown_module(mod): > + py.code.unpatch_builtins(assertion=True, compile=False) > + > + > +def test_assert_simple(): > + # Simply test that this way of testing works > + a = 0 > + b = 1 > + r = interpret('assert a == b', getframe()) > + assert r == 'assert 0 == 1' > + > + > +def test_assert_list(): > + r = interpret('assert [0, 1] == [0, 2]', getframe()) > + msg = ('assert [0, 1] == [0, 2]\n' > + ' First differing item 1: 1 != 2\n' > + ' - [0, 1]\n' > + ' ? ^\n' > + ' + [0, 2]\n' > + ' ? ^') > + print r > + assert r == msg > + > + > +def test_assert_string(): > + r = interpret('assert "foo and bar" == "foo or bar"', getframe()) > + msg = ("assert 'foo and bar' == 'foo or bar'\n" > + " - foo and bar\n" > + " ? ^^^\n" > + " + foo or bar\n" > + " ? ^^") > + print r > + assert r == msg > + > + > +def test_assert_multiline_string(): > + a = 'foo\nand bar\nbaz' > + b = 'foo\nor bar\nbaz' > + r = interpret('assert a == b', getframe()) > + msg = ("assert 'foo\\nand bar\\nbaz' == 'foo\\nor bar\\nbaz'\n" > + ' foo\n' > + ' - and bar\n' > + ' + or bar\n' > + ' baz') > + print r > + assert r == msg > + > + > +def test_assert_dict(): > + a = {'a': 0, 'b': 1} > + b = {'a': 0, 'c': 2} > + r = interpret('assert a == b', getframe()) > + msg = ("assert {'a': 0, 'b': 1} == {'a': 0, 'c': 2}\n" > + " - {'a': 0, 'b': 1}\n" > + " ? ^ ^\n" > + " + {'a': 0, 'c': 2}\n" > + " ? ^ ^") > + print r > + assert r == msg > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -- From Ronny.Pfannschmidt at gmx.de Mon Aug 16 13:29:13 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Mon, 16 Aug 2010 13:29:13 +0200 Subject: [py-dev] Advanced assert equal In-Reply-To: <20100816002557.GA12335@laurie.devork.be> References: <20100816002557.GA12335@laurie.devork.be> Message-ID: <1281958153.12940.5.camel@klappe2> On Mon, 2010-08-16 at 01:25 +0100, Floris Bruynooghe wrote: > Hi > > Ever since unittest grew it's .assertSequenceEqual() and > .assertMultilineEqual() I've been jealous of it. So this weekend I've > looked into the py.test code and made an attempt at getting this into > my favourite testing tool. > > The attached patch makes compare equal a special case and checks if > the two arguments to it are both a list, text or dict and tries to > generate a nicer explanation text for them. The patch is more like a > proof of concept then a final implementation, I may have done some > very strange or silly things as I'm not familiar with the code. It > would be great to get feedback, both on the general concept and the > actual implementation (particularly note the way I had to hack > _format_explanation() in assertion.py). > I think it will be helpful to have some kind of hook to add more explain-functions In particular, cause there are many more build-in types to manage, and i have at least 2 projects where custom compare-explain is helpfull another wishlist item i see is the rest of rich compare i.e. <, >, <=, >=, != > Some of the rough edges I can think off right now: (i) no idea how > comparisons and nested calls work together, (ii) no attempt is made to > limit the output from difflib so the screen doesn't get flooded. > There's probably many more. > > I hope this can be useful > Floris > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From flub at devork.be Mon Aug 16 15:51:54 2010 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 16 Aug 2010 14:51:54 +0100 Subject: [py-dev] Advanced assert equal In-Reply-To: <1281958153.12940.5.camel@klappe2> References: <20100816002557.GA12335@laurie.devork.be> <1281958153.12940.5.camel@klappe2> Message-ID: <20100816135154.GA24594@laurie.devork.be> On Mon, Aug 16, 2010 at 01:29:13PM +0200, Ronny Pfannschmidt wrote: > On Mon, 2010-08-16 at 01:25 +0100, Floris Bruynooghe wrote: > > The attached patch makes compare equal a special case and checks if > > the two arguments to it are both a list, text or dict and tries to > > generate a nicer explanation text for them. The patch is more like a > > proof of concept then a final implementation, I may have done some > > very strange or silly things as I'm not familiar with the code. It > > would be great to get feedback, both on the general concept and the > > actual implementation (particularly note the way I had to hack > > _format_explanation() in assertion.py). > > I think it will be helpful to have some kind of hook to add more > explain-functions Adding hooks should be possible, looking at all the .visit_*() functions it would seem only one hook is required, unless separate hooks for each rich compare operator are deemed useful. The trickiest bit I think is how to produce multiline explanations. _format_explanation() concatenates all newlines. Having only special cases for \n{ and \n} which is used by .visit_Call() to force nested and indented newlines. In the patch I added \n== for this but a more general one is probably required, something like \n> or \n~ could work I guess. This could be completely hidden from the hook however, by returning a list for each line to be printed, the caller of the hook would then join these up correctly so that _format_explanation() will add newlines and indentation correctly. A possible api for the hook could be: def pytest_assert_compare(op, left, right, left_repr, right_repr): """Customise compare Return None for no custom compare, otherwise return a list of strings. The strings will be joined by newlines but any newlines *in* a string will be escaped. """ pass I guess the reprs are not really necessary if there's an easy way to make them. It's just that I used them in my original patch. Another option is to encapsulate the arguments into an object that also knows how the builtin types and operators are compared, something like: class CompareItem(object): def __init__(self, op, left, right, ...): self.op = op self.left = left self.right = right ... def default(self): if type(self.left) != type(self.right): return None if self.op == '==': if isinstance(self.left, (list, tuple)): return self.sequence_equal() elif isinstance(self.left, basestring): return self.string_equal() ... elif self.op == '!=': ... def sequence_equal(self): pass def string_equal(self): pass ... This would allow pytest_assert_compare() to use those methods as part of the implementation. There's also the question of who should truncate large amounts of data (e.g. screenfulls of diffs): the hook itself, the caller of the hook or _format_explanation()? Probably one of the first two to get rid of the memory usage as soon as possible. > In particular, cause there are many more build-in types to manage, > and i have at least 2 projects where custom compare-explain is helpfull > > another wishlist item i see is the rest of rich compare > i.e. <, >, <=, >=, != Sure, all builtin types and operators should ideally be supported by default as best as possible. I just started with some I wanted most. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From Ronny.Pfannschmidt at gmx.de Tue Aug 17 12:54:53 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 17 Aug 2010 12:54:53 +0200 Subject: [py-dev] [proposal] extended information collection Message-ID: <1282042493.12940.16.camel@klappe2> Recently i noticed a new need. The collecting/asserting various kinds of data during a testrun. I noticed the following (partially hpyothetical) cases in various projects * pip records file changes to the virtualenv it makes for testing * pypy records jit compile results/dumps * jinja2 should record asts and the compile results of templates it tests * the logcapture extension * the build-in io captureing i think i would be helpful/practical to have a common way to record those things and navigate them in extended test reporters adding propper timing information to lines/data could help even more (for having a common timeline of different event sources) since this topic is rather hard to manage i hope for some dicussion on the needs others see, as well as if/how this could add more to the testing regards Ronny From Florian.Bauer at diasemi.com Fri Aug 20 15:49:35 2010 From: Florian.Bauer at diasemi.com (Florian Bauer) Date: Fri, 20 Aug 2010 15:49:35 +0200 Subject: [py-dev] py.test doesn't like functools.partial wrappers Message-ID: <34CD30EDA5E4CB4AAA2501EB1ED319E102D705B6@charlie.diasemi.com> Hi all, I'm trying to make an existing testsuite py.test compatible. At the moment, we use nose as test runner. I stumbled upon the following (simplified example). from functools import partial def forall_cases(cases): def decorate(testfn): def gen(): for case in cases: description = case[0] args = case[1:] func = partial(testfn, *args) func.description = description yield func, gen.__name__= 'test_%s_' % testfn.__name__ # # inject the generator into the module testfn came from # gen.__module__ = testfn.__module__ return gen return decorate @forall_cases([['case1', 1, 1], ['case2', 2, 2],['case3', 3, 4]]) def test_equal(a, b): assert a == b If I run this with nosetests, I get 3 test cases, 2 pass and 1 fail. py.test tries to call inspect.getsource on the partial object, which results in a TypeError: $ py.test -v ... INTERNALERROR> INTERNALERROR> object = INTERNALERROR> INTERNALERROR> def getfile(object): INTERNALERROR> """Work out which source or compiled file an INTERNALERROR> object was d efined in.""" ... INTERNALERROR> > raise TypeError('arg is not a module, class, method, ' INTERNALERROR> 'function, traceback, frame, or INTERNALERROR> code obje ct') INTERNALERROR> E TypeError: arg is not a module, class, method, function, traceback, frame, or code object INTERNALERROR> INTERNALERROR> object = INTERNALERROR> INTERNALERROR> c:\Python26\lib\inspect.py:418: TypeError This seems to be a documented limitation of inspect.getsource (functools.partial returns a partial object, not a function). I can fix this by replacing functools.partial with code given as roughly equivalent in the python documentation: def partial(func, *args, **keywords): def newfunc(*fargs, **fkeywords): newkeywords = keywords.copy() newkeywords.update(fkeywords) return func(*(args + fargs), **newkeywords) newfunc.func = func newfunc.args = args newfunc.keywords = keywords return newfunc But I was certainly not expecting this. The default behavior of py.test is even more puzzling, as it fails silently: fbauer at MN-ENG-LT-FB ~/My Documents/Software/pytestbug1 $ py.test ============================= test session starts ============================= platform win32 -- Python 2.6.5 -- pytest-1.3.3 test path 1: c:\docume~1\fbauer\My Documents\Software\pytestbug1 test_case.py .. In my actual code, the forall_cases decorator lives in a utils module. Is there a conditional flag that can tell me whether I'm running under py.test? Then I could start experimenting with parametric tests using funcargs, while keeping the test suite runnable with nose at the moment. I still have a bug in the test suite, as I have tests passing under nose that fail under py.test... Best regards, Florian _______________________________________________________________________________________ Dialog Semiconductor GmbH Neue Str. 95 D-73230 Kirchheim Managing Director: Dr. Jalal Bagherli Chairman of the Supervisory Board: Gregorio Reyes Commercial register: Amtsgericht Stuttgart: HRB 231181 UST-ID-Nr. DE 811121668 Legal Disclaimer: This e-mail communication (and any attachment/s) is confidential and contains proprietary information, some or all of which may be legally privileged. It is intended solely for the use of the individual or entity to which it is addressed. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. From richard.m.tew at gmail.com Sat Aug 21 03:20:44 2010 From: richard.m.tew at gmail.com (Richard Tew) Date: Sat, 21 Aug 2010 09:20:44 +0800 Subject: [py-dev] Greenlet Message-ID: Hi, In the past, I believe py included the greenlet c extension source code. I am not sure that you do anymore, or how this affects you now that it may only relate to older releases of your source code, but I would like to mention that the py library as far as I can tell effectively relicensed Python licensed code under the MIT license. Please correct me if I am wrong. http://codespeak.net/svn/py/branch/py-compat-2.5.2/py/c-extension/greenlet/ The files sourced from Stackless presumably by Christian for use with Armin in the creation of greenlet, were contributed to Stackless and subject to contributions from multiple parties, under the Python license. I am pretty sure that Christian did not, and was more than likely unable to relicense them to the MIT license on behalf of those parties. Anyway, I've brought this up with the greenlet project here: http://bitbucket.org/ambroff/greenlet/ And since that was spun off the py lib, I thought I would mention it here. Cheers, Richard. From Ronny.Pfannschmidt at gmx.de Tue Aug 24 08:43:42 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 24 Aug 2010 08:43:42 +0200 Subject: [py-dev] py.test doesn't like functools.partial wrappers In-Reply-To: <34CD30EDA5E4CB4AAA2501EB1ED319E102D705B6@charlie.diasemi.com> References: <34CD30EDA5E4CB4AAA2501EB1ED319E102D705B6@charlie.diasemi.com> Message-ID: <1282632222.17701.4.camel@klappe2> On Fri, 2010-08-20 at 15:49 +0200, Florian Bauer wrote: > Hi all, > > I'm trying to make an existing testsuite py.test compatible. > At the moment, we use nose as test runner. I stumbled upon the following > (simplified example). > > from functools import partial > > def forall_cases(cases): > def decorate(testfn): > def gen(): > for case in cases: > description = case[0] > args = case[1:] > func = partial(testfn, *args) > func.description = description > yield func, > gen.__name__= 'test_%s_' % testfn.__name__ > # > # inject the generator into the module testfn came from > # > gen.__module__ = testfn.__module__ > return gen > return decorate > > > @forall_cases([['case1', 1, 1], ['case2', 2, 2],['case3', 3, 4]]) def > test_equal(a, b): > assert a == b > > > If I run this with nosetests, I get 3 test cases, 2 pass and 1 fail. > > py.test tries to call inspect.getsource on the partial object, which > results in a TypeError: > > $ py.test -v > ... > INTERNALERROR> > INTERNALERROR> object = > INTERNALERROR> > INTERNALERROR> def getfile(object): > INTERNALERROR> """Work out which source or compiled file an > INTERNALERROR> object was d > efined in.""" > ... > INTERNALERROR> > raise TypeError('arg is not a module, class, > method, ' > INTERNALERROR> 'function, traceback, frame, or > INTERNALERROR> code obje > ct') > INTERNALERROR> E TypeError: arg is not a module, class, method, > function, > traceback, frame, or code object > INTERNALERROR> > INTERNALERROR> object = > INTERNALERROR> > INTERNALERROR> c:\Python26\lib\inspect.py:418: TypeError > > This seems to be a documented limitation of inspect.getsource > (functools.partial returns a partial object, not a function). > I can fix this by replacing functools.partial with code given as roughly > equivalent in the python documentation: > > def partial(func, *args, **keywords): > def newfunc(*fargs, **fkeywords): > newkeywords = keywords.copy() > newkeywords.update(fkeywords) > return func(*(args + fargs), **newkeywords) > newfunc.func = func > newfunc.args = args > newfunc.keywords = keywords > return newfunc > > But I was certainly not expecting this. The default behavior of py.test > is even more puzzling, as it fails silently: > > fbauer at MN-ENG-LT-FB ~/My Documents/Software/pytestbug1 $ py.test > ============================= test session starts > ============================= platform win32 -- Python 2.6.5 -- > pytest-1.3.3 test path 1: c:\docume~1\fbauer\My > Documents\Software\pytestbug1 > > test_case.py .. that looks like 2 bugs cooperating for certain confusion not being able to get the source causes pytest to internally fail and those seem to be failing i'll try to make minimal replications for the pytest testsuite as i cant tell from your trace where exactly the issue orginates > > In my actual code, the forall_cases decorator lives in a utils module. > Is there a conditional flag that can tell me whether I'm running under > py.test? Then I could start experimenting with parametric tests using > funcargs, while keeping the test suite runnable with nose at the moment. > I still have a bug in the test suite, as I have tests passing under nose > that fail under py.test... > im not sure of such a flag, you could use the pytest_configure hook in a conftest to set up certain globals same goes for the per module setup hooks also you could try to use the collect hooks to strip your decorated testcases of the decorator Regards, Ronny From Ronny.Pfannschmidt at gmx.de Tue Aug 24 08:49:26 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 24 Aug 2010 08:49:26 +0200 Subject: [py-dev] py.test doesn't like functools.partial wrappers In-Reply-To: <34CD30EDA5E4CB4AAA2501EB1ED319E102D705B6@charlie.diasemi.com> References: <34CD30EDA5E4CB4AAA2501EB1ED319E102D705B6@charlie.diasemi.com> Message-ID: <1282632566.17701.8.camel@klappe2> on a sidenote, according to the tests one is supposed to just use the name/description before the function in the result tuple i.e. yield name, func, ... instead of yield partial_with_desc, ... i suppose setting a global from conftest could help if you just switch cases after that on pytest you could replace your decoration with a py.test.mark.forall or something like that Regards Ronny