From holger at merlinux.eu Wed Sep 3 16:33:46 2008 From: holger at merlinux.eu (holger krekel) Date: Wed, 3 Sep 2008 16:33:46 +0200 Subject: [py-dev] showing tracebacks / failure handling (was: Re: [arigo@codespeak.net: [py-svn] r57790 - in py/trunk/py: code test test/report test/report/testing] In-Reply-To: <20080903101008.GA5776@code0.codespeak.net> References: <20080903083052.GT9619@trillke.net> <20080903101008.GA5776@code0.codespeak.net> Message-ID: <20080903143346.GW9619@trillke.net> Hi Armin, (CC to py-dev) On Wed, Sep 03, 2008 at 12:10 +0200, Armin Rigo wrote: > Hi Holger, > > On Wed, Sep 03, 2008 at 10:30:53AM +0200, holger krekel wrote: > > i think the/a test should go to > > > > py/code/testing/test_excinfo.py:430 > > True, I can add a test there. But I had to change a few small things in > py/test/ too, so the existing test should not be removed. ok let's have both. > > also, i wonder if "tb=crash" would be useful so that only the > > crashing function is shown. > > Not for me. I don't use tb=short very often, but when I use it it's for > cases where PyPy tests spew tracebacks with a hundred entries. I use > tb=no a lot though. do you mean a few hundred failures? If you mean one failure with a few hundred entries than tb=crash would just show one entry. maybe we better sort this out on IRC :) > One thing I'm really missing from the trunk py.test output is the name > of the failing tests. I know it's often the name of the first function > in the traceback but this is not always reliable (e.g. with generated > code or broken .pyc files). My use case here is again PyPy: often, you > have 5 failures that each spew a lot of stdout and a long traceback. > Figuring out which 5 tests failed takes quite a while. In my > rxvt-terminal-only conftest, I can double-click on the final line that > says "5 failures" and see a list of the names of the failing tests. In > my opinion it would even be useful to always list these names in the > final line, e.g. looking like: > > ============= 2 failures: test_foo, test_bar =============== names are usually much longer, particularly if you include class and module name. > with something reasonable if there are too many failures, e.g.: > > ====== 17 failures: test_foo, test_bar, test_baz, ... ====== > > the point being that it's easy to re-run "py.test -k test_foo" to > reproduce just the first failure, which is often the most relevant one, > without having to use "py.test -x" which can take a long time to run > before it hits the failure. i see the point. hum, we could show a summary (by default or through option) after the tracebacks similar to what you you see with "--verbose", e.g. with curdir=py/doc/ and py.test --verbose pytest/failure_demo.py you currently get pytest/failure_demo.py:15: TestFailing.test_simple FAIL pytest/failure_demo.py:23: TestFailing.test_simple_multiline FAIL pytest/failure_demo.py:28: TestFailing.test_not FAIL pytest/failure_demo.py:33: TestFailing.test_complex_error FAIL pytest/failure_demo.py:40: TestFailing.test_z1_unpack_error FAIL pytest/failure_demo.py:44: TestFailing.test_z2_type_error FAIL ... But a more fundamental solution is to have py.test store info about the last test run (by option and/or some ENV variable) and have options/ways to show and run the last failing tests. The latter is also what we want for the new htmlconftest so i suggest we maybe first concentrate on this. in any case i'd be grateful if you put/copied your hacks around py.test to directories in contrib (maybe with at least one functional test) so that it'seasier to work toegether and ensure that things work after refactorings. best & thanks, holger From arigo at tunes.org Wed Sep 3 19:04:44 2008 From: arigo at tunes.org (Armin Rigo) Date: Wed, 3 Sep 2008 19:04:44 +0200 Subject: [py-dev] showing tracebacks / failure handling (was: Re: [arigo@codespeak.net: [py-svn] r57790 - in py/trunk/py: code test test/report test/report/testing] In-Reply-To: <20080903143346.GW9619@trillke.net> References: <20080903083052.GT9619@trillke.net> <20080903101008.GA5776@code0.codespeak.net> <20080903143346.GW9619@trillke.net> Message-ID: <20080903170442.GA14244@code0.codespeak.net> Hi Holger, On Wed, Sep 03, 2008 at 04:33:46PM +0200, holger krekel wrote: > in any case i'd be grateful if you put/copied your hacks > around py.test to directories in contrib (maybe with at > least one functional test) so that it'seasier to work > toegether and ensure that things work after refactorings. My conftest monkey-patches completely internal details around, so keeping it working would just prevent any refactoring. I'm ok with having to throw my conftest mostly away and start again from scratch after a big refactoring. A bientot, Armin. From holger at merlinux.eu Wed Sep 3 19:37:50 2008 From: holger at merlinux.eu (holger krekel) Date: Wed, 3 Sep 2008 19:37:50 +0200 Subject: [py-dev] showing tracebacks / failure handling (was: Re: [arigo@codespeak.net: [py-svn] r57790 - in py/trunk/py: code test test/report test/report/testing] In-Reply-To: <20080903170442.GA14244@code0.codespeak.net> References: <20080903083052.GT9619@trillke.net> <20080903101008.GA5776@code0.codespeak.net> <20080903143346.GW9619@trillke.net> <20080903170442.GA14244@code0.codespeak.net> Message-ID: <20080903173750.GY9619@trillke.net> Hi Armin, On Wed, Sep 03, 2008 at 19:04 +0200, Armin Rigo wrote: > Hi Holger, > > On Wed, Sep 03, 2008 at 04:33:46PM +0200, holger krekel wrote: > > in any case i'd be grateful if you put/copied your hacks > > around py.test to directories in contrib (maybe with at > > least one functional test) so that it'seasier to work > > toegether and ensure that things work after refactorings. > > My conftest monkey-patches completely internal details around, so > keeping it working would just prevent any refactoring. I'm ok with > having to throw my conftest mostly away and start again from scratch > after a big refactoring. well, i regard your rxvt and particularly your htmlconftest hacks as valid extensions and would like to improve py.test such that you do not need to patch internal details. holger From orestis at orestis.gr Sat Sep 20 23:59:19 2008 From: orestis at orestis.gr (Orestis Markou) Date: Sat, 20 Sep 2008 22:59:19 +0100 Subject: [py-dev] Information for unittest refugees Message-ID: <29839E33-8487-4E44-B133-6E76877CF4ED@orestis.gr> After attending Holger's talk at PyCon UK about py.test, I decided to try it out. I'm using a recent svn checkout. I had a bunch of tests (around 50 of them), all inheriting from unittest. I expected that they would work out of the box with py.test, and was surprised when they weren't. I tried adding the py_unittest conftest.py file to my project, but I got this traceback: Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/ Current/bin/py.test", line 8, in load_entry_point('py==1.0.0a1', 'console_scripts', 'py.test')() File "/Users/orestis/Developer/pylib/py/cmdline/pytest.py", line 5, in main py.test.cmdline.main() File "/Users/orestis/Developer/pylib/py/test/cmdline.py", line 12, in main config.parse(args) File "/Users/orestis/Developer/pylib/py/test/config.py", line 46, in parse self._conftest.setinitial(args) File "/Users/orestis/Developer/pylib/py/test/conftesthandle.py", line 30, in setinitial self._path2confmods[None] = self.getconftestmodules(anchor) File "/Users/orestis/Developer/pylib/py/test/conftesthandle.py", line 44, in getconftestmodules clist.append(importconfig(conftestpath)) File "/Users/orestis/Developer/pylib/py/test/conftesthandle.py", line 74, in importconfig mod = configpath.pyimport() File "/Users/orestis/Developer/pylib/py/path/local/local.py", line 413, in pyimport mod = __import__(modname, None, None, ['__doc__']) File "/Users/orestis/Developer/pysmell/conftest.py", line 52, in class UnitTestFunction(py.__.test.item.Function): AttributeError: 'module' object has no attribute 'item' I expected that the naming conventions would be enough, but I was surprised again by the fact that py.test requires "test_" methods rather than just "test" methods. Fixing that, I was again surprised that my tests still weren't running, and after some guesswork realized that the class they were in didn't start with "Test". I suggest that you add this crucial information in a more clear way (it is there, but not very discoverable) or even better, integrate the py_unittest collection into py.test - this way people can try out their old tests with py.test, rather than having to convert everything. Best regards, Orestis Markou -- orestis at orestis.gr http://orestis.gr/ From simon at arrowtheory.com Sun Sep 21 00:29:56 2008 From: simon at arrowtheory.com (Simon Burton) Date: Sat, 20 Sep 2008 18:29:56 -0400 Subject: [py-dev] py.test problem Message-ID: <20080920182956.934c637d.simon@arrowtheory.com> I have hit this problem several times, when an assertion fails using py.test i get AttributeError: << 'frame' object has no attribute 'eval' with no stack trace of where in py.test this occured. (only the stack trace of my test code). Any ideas ? It looks like it has something to do with the magic that prints out the values of expressions in the assertion. I just did svn up on my py.test, it did not help. Simon. From holger at merlinux.eu Sun Sep 21 08:23:40 2008 From: holger at merlinux.eu (holger krekel) Date: Sun, 21 Sep 2008 08:23:40 +0200 Subject: [py-dev] py.test problem In-Reply-To: <20080920182956.934c637d.simon@arrowtheory.com> References: <20080920182956.934c637d.simon@arrowtheory.com> Message-ID: <20080921062340.GA22099@trillke.net> Hi Simon, On Sat, Sep 20, 2008 at 18:29 -0400, Simon Burton wrote: > I have hit this problem several times, when an assertion fails using py.test i get > > AttributeError: << 'frame' object has no attribute 'eval' > > with no stack trace of where in py.test this occured. (only the stack trace of my test code). > > Any ideas ? hum, can't remember seeing this one. Do you have an example? If not can you attach a traceback next time? > It looks like it has something to do with the magic that prints out the values of expressions > in the assertion. You can use --nomagic to turn it off. > I just did svn up on my py.test, it did not help. that was 'dist'? svn/py/trunk has a lot of cleaned up code which might help remedy the problem. sorry that i can't be more immediately helpful, thanks, holger From holger at merlinux.eu Sun Sep 21 10:23:56 2008 From: holger at merlinux.eu (holger krekel) Date: Sun, 21 Sep 2008 10:23:56 +0200 Subject: [py-dev] Information for unittest refugees In-Reply-To: <29839E33-8487-4E44-B133-6E76877CF4ED@orestis.gr> References: <29839E33-8487-4E44-B133-6E76877CF4ED@orestis.gr> Message-ID: <20080921082356.GB22099@trillke.net> Hi Orestis, thanks for your mail and sorry for the hassle - to be honest i haven't cared much for integrating unittests myself yet. But i try to improve now :) On Sat, Sep 20, 2008 at 22:59 +0100, Orestis Markou wrote: > After attending Holger's talk at PyCon UK about py.test, I decided to > try it out. I'm using a recent svn checkout. > > I had a bunch of tests (around 50 of them), all inheriting from > unittest. I expected that they would work out of the box with py.test, > and was surprised when they weren't. I tried adding the py_unittest > conftest.py file to my project, but I got this traceback: > > Traceback (most recent call last): > File "/opt/local/Library/Frameworks/Python.framework/Versions/ > Current/bin/py.test", line 8, in > load_entry_point('py==1.0.0a1', 'console_scripts', 'py.test')() > File "/Users/orestis/Developer/pylib/py/cmdline/pytest.py", line 5, > in main > py.test.cmdline.main() > File "/Users/orestis/Developer/pylib/py/test/cmdline.py", line 12, > in main > config.parse(args) > File "/Users/orestis/Developer/pylib/py/test/config.py", line 46, > in parse > self._conftest.setinitial(args) > File "/Users/orestis/Developer/pylib/py/test/conftesthandle.py", > line 30, in setinitial > self._path2confmods[None] = self.getconftestmodules(anchor) > File "/Users/orestis/Developer/pylib/py/test/conftesthandle.py", > line 44, in getconftestmodules > clist.append(importconfig(conftestpath)) > File "/Users/orestis/Developer/pylib/py/test/conftesthandle.py", > line 74, in importconfig > mod = configpath.pyimport() > File "/Users/orestis/Developer/pylib/py/path/local/local.py", line > 413, in pyimport > mod = __import__(modname, None, None, ['__doc__']) > File "/Users/orestis/Developer/pysmell/conftest.py", line 52, in > > class UnitTestFunction(py.__.test.item.Function): > AttributeError: 'module' object has no attribute 'item' i think this is a mismatch between the py lib checkout and the conftest. If you checkout http://codespeak.net/svn/py/trunk * install it e.g. by doing "python setup.py develop" and * put contrib/py_unittest/conftest.py into pysmell then it should work fine now. I actually just tested with your current GIT version and all but two tests fail (they look like expected failures). > I expected that the naming conventions would be enough, but I was > surprised again by the fact that py.test requires "test_" methods > rather than just "test" methods. Fixing that, I was again surprised > that my tests still weren't running, and after some guesswork realized > that the class they were in didn't start with "Test". hum, methods/functions only need to start with "test" in any case. If not there is a bug and i am happy about an example. unittest-collected classes don't need to follow a naming convention - they just need to subclass TestCase as usual. py.test-collected classes indeed need to start with "Test". So i am a bit confused what your exact problem actually was here. Btw, for debugging, "py.test --collectonly" is helpful. > I suggest that you add this crucial information in a more clear way > (it is there, but not very discoverable) or even better, integrate the > py_unittest collection into py.test - this way people can try out > their old tests with py.test, rather than having to convert everything. good point. So far i was hesitant with promoting it because i wanted to spend a bit more time into docs and an infrastructure that helps with checking that things are consistent. It apparently gets about time now :) best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From orestis at orestis.gr Sun Sep 21 12:41:39 2008 From: orestis at orestis.gr (Orestis Markou) Date: Sun, 21 Sep 2008 11:41:39 +0100 Subject: [py-dev] Information for unittest refugees In-Reply-To: <20080921082356.GB22099@trillke.net> References: <29839E33-8487-4E44-B133-6E76877CF4ED@orestis.gr> <20080921082356.GB22099@trillke.net> Message-ID: <158DF2B2-FE29-47AA-9F21-F2CB8E247941@orestis.gr> > i think this is a mismatch between the py lib checkout and > the conftest. If you checkout > > http://codespeak.net/svn/py/trunk > > * install it e.g. by doing "python setup.py develop" and > * put contrib/py_unittest/conftest.py into pysmell > > then it should work fine now. I actually just tested > with your current GIT version and all but two tests fail > (they look like expected failures). > Ah, I was following your slides and checkout out the py_unittest directory directly - copying contrib/py_unittest/conftest.py worked! >> I expected that the naming conventions would be enough, but I was >> surprised again by the fact that py.test requires "test_" methods >> rather than just "test" methods. Fixing that, I was again surprised >> that my tests still weren't running, and after some guesswork >> realized >> that the class they were in didn't start with "Test". > > hum, methods/functions only need to start with "test" in any case. > If not there is a bug and i am happy about an example. Sorry, I was confused because I didn't prepend Test to my classes. Indeed no underscore was needed. > > > unittest-collected classes don't need to follow a naming > convention - they just need to subclass TestCase as usual. > > py.test-collected classes indeed need to start with "Test". > > So i am a bit confused what your exact problem actually was > here. Btw, for debugging, "py.test --collectonly" is helpful. My problem was that I had a lot of test_* methods in a class that didn't start with Test (not inheriting from unittest). I couldn't find that bit in the documentation that mentioned that - I will try to write up a small py.test in 1 minute document that you could use. >> I suggest that you add this crucial information in a more clear way >> (it is there, but not very discoverable) or even better, integrate >> the >> py_unittest collection into py.test - this way people can try out >> their old tests with py.test, rather than having to convert >> everything. > > good point. So far i was hesitant with promoting it > because i wanted to spend a bit more time into docs and > an infrastructure that helps with checking that things > are consistent. It apparently gets about time now :) I suggest that version 1 should have this bit in, it will help a lot of people to give it a casual go :) Regards, Orestis From nshepperd at gmail.com Mon Sep 22 03:53:48 2008 From: nshepperd at gmail.com (Neil Shepperd) Date: Mon, 22 Sep 2008 11:53:48 +1000 Subject: [py-dev] py.test --looponfailing (SVN r58313) AssertionError Message-ID: <1222048428.26602.15.camel@neil> Hi all, Py.test dies with an assertion error when I change the name of a failing test, or move the file or directory it's in. It looks like it's just trying to get the result of the test without checking whether it exists first. This is with the HEAD revision. Neil ---------------- Example: [test_blah.py] def test_blah(): assert False $ py.test --looponfailing [edit test_blah.py] -def test_blah(): +def test_x(): Traceback (most recent call last): ... AssertionError: Collector does not have name 'test_blah' existing names are: ['test_blah.py'] From holger at merlinux.eu Mon Sep 22 15:16:43 2008 From: holger at merlinux.eu (holger krekel) Date: Mon, 22 Sep 2008 15:16:43 +0200 Subject: [py-dev] py.test --looponfailing (SVN r58313) AssertionError In-Reply-To: <1222048428.26602.15.camel@neil> References: <1222048428.26602.15.camel@neil> Message-ID: <20080922131643.GG19528@trillke.net> Hi Neil, thanks for your good report! I just committed a fix in 58319 let me know if it solves the issue. holger On Mon, Sep 22, 2008 at 11:53 +1000, Neil Shepperd wrote: > Hi all, > > Py.test dies with an assertion error when I change the name > of a failing test, or move the file or directory it's in. It > looks like it's just trying to get the result of the test without > checking whether it exists first. > > This is with the HEAD revision. > > Neil > > ---------------- > Example: > > [test_blah.py] > def test_blah(): > assert False > > $ py.test --looponfailing > > [edit test_blah.py] > -def test_blah(): > +def test_x(): > > Traceback (most recent call last): > ... > AssertionError: Collector does not have name > 'test_blah' existing names are: ['test_blah.py'] > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org