From pedronis at openend.se Tue Apr 6 17:36:24 2010 From: pedronis at openend.se (Samuele Pedroni) Date: Tue, 06 Apr 2010 17:36:24 +0200 Subject: [py-dev] ANN: oejskit 0.8.8 JavaScript in-browser testing with py.test plugin and unittest.py glue Message-ID: <4BBB54F8.4060500@openend.se> Hi, I just pushed a new release of OE jskit 0.8.8 to PyPI. Main points of interest: * the code to check for the presence of browsers locally has been improved, browser specifications can now with confidence list absent browsers and the respective runs/tests will be skipped * added a workaround to a bug in Firefox 3.5 that interferes with the global variable leak detection code, this is simply turned off for FF3.5. The bug itself is fixed in FF3.6 About OE jskit: jskit contains infrastructure and in particular a py.test plugin to enable running unit tests for JavaScript code inside browsers. It contains also glue code to run JavaScript tests from unittest.py based test suites. The approach also enables to write integration tests such that the JavaScript code is tested against server-side Python code mocked as necessary. Any server-side framework that can already be exposed through WSGI can play. The plugin requires py.test 1.1.1 and should also work with current trunk. More information and downloading at: http://pypi.python.org/pypi/oejskit including a changelog, documentation and the talk I gave at Europython 2009. jskit was initially developed by Open End AB and is released under the MIT license. In various incarnations it has been in use and useful at Open End for more than two years, we are quite happy to share it. Samuele Pedroni for Open End _______________________________________________ py-dev mailing list py-dev at codespeak.net http://codespeak.net/mailman/listinfo/py-dev From simongsmith at gmail.com Wed Apr 7 22:42:39 2010 From: simongsmith at gmail.com (simongsmith at gmail.com) Date: Wed, 07 Apr 2010 20:42:39 -0000 Subject: [py-dev] using oejskit to test hosted html Message-ID: I've been exploring the examples of using oejskit (from [http://www2.openend.se/~pedronis/oejskit/talk/examples.tgz: ]) and they work fine for me. I wanted to change the test_open_eval.py to access the same foo.html via an apache website I'm hosting (it is static in the original). I'm envoking with "py.test test" and am using python2.6, py.test 1.2.1 and oejskit 0.8.8 on a Mac running OS X. Original code (test/test_open_eval.py) from oejskit.testing import BrowserTestClass, JsFailed class TestOpenEval(BrowserTestClass): jstests_browser_kind = 'supported' def test_open_eval(self): pg = self.open('/static/foo.html') title = pg.eval('document.title') assert title == 'Foo' I changed the open call: pg = self.open('http://mysite.net/foo.html') When I re-run the test, the foo.html is loaded into the browser that is launched as a result of the test, and it is identical (via "View Frame Source") as the original, but I get the following error: self = js = 'document.title', variant = 'eval' def eval(self, js, variant='eval'): outcome = self._execute(variant, js, self.root, self.timeout) if outcome.get('error'): > raise JsFailed('[%s] %s' % (self.label, js), outcome['error']) E JsFailed: [[http://ctier.arcode.com:8888/foo.html: ]] document.title: FAILED: E message: Result of expression 'frameWin.eval' [undefined] is not a function. E line: 169 E sourceId: 457014824 E sourceURL: [http://localhost:62737/browser_testing/rt/InBrowserTesting.js: ] E expressionBeginOffset: 4985 E expressionCaretOffset: 4998 E expressionEndOffset: 5004 E name: TypeError ../../../arcode/python/arcode/js- infrastructure/oejskit/browser_ctl.py:263: JsFailed I've dug around in the oejskit code and I don't understand what's going wrong and I can't find any documentation on this? has anyone successfully used oejskit to assert against html hosted on a remote apache? Or does anyone have any suggestions as to what I might be doing incorrectly? foo.html follows below. Thanks!!! Simon Smith [simongsmith at gmail.com: ] foo.html: Foo<title> <script> </script> </head> <body> </body> </html> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20100407/26b05693/attachment.html> From pedronis at openend.se Thu Apr 8 13:47:33 2010 From: pedronis at openend.se (Samuele Pedroni) Date: Thu, 08 Apr 2010 13:47:33 +0200 Subject: [py-dev] using oejskit to test hosted html In-Reply-To: <TGmx3i_RRTv0XKAG6Ejdc-2IGXlQdAvMVKkYvDbAjhzwtsJFc@smtp.gmail.com> References: <TGmx3i_RRTv0XKAG6Ejdc-2IGXlQdAvMVKkYvDbAjhzwtsJFc@smtp.gmail.com> Message-ID: <4BBDC255.3020206@openend.se> simongsmith at gmail.com wrote: > > I changed the open call: > > > > pg = self.open('http://mysite.net/foo.html') > > When I re-run the test, the foo.html is loaded into the browser that > is launched as a result of the test, and it is identical (via "View > Frame Source") as the original, but I get the following error: > > > > self = <oejskit.browser_ctl.PageContext object at 0x156dc90> > js = 'document.title', variant = 'eval' > > def eval(self, js, variant='eval'): > outcome = self._execute(variant, js, self.root, self.timeout) > if outcome.get('error'): > > raise JsFailed('[%s] %s' % (self.label, js), outcome['error']) > E JsFailed: [http://ctier.arcode.com:8888/foo.html] > document.title: FAILED: > E message: Result of expression 'frameWin.eval' [undefined] > is not a function. > E line: 169 > E sourceId: 457014824 > E sourceURL: > http://localhost:62737/browser_testing/rt/InBrowserTesting.js > E expressionBeginOffset: 4985 > E expressionCaretOffset: 4998 > E expressionEndOffset: 5004 > E name: TypeError > > ../../../arcode/python/arcode/js-infrastructure/oejskit/browser_ctl.py:263: > JsFailed > > > > > Hi, there are two levels of possible issues here: - in some browsers it seems (as foo.html does) one needs at least an empty script tag to have eval() available on the window object - more importantly while it is possible to use oejskit to do functional testing (we have used it here that way) because of javascript security the tested pages/app and testing infrastructure need to come from the same domain, this can be arranged with proxying but there's no example right now how to do this and this depends on how the pages/app themselves are served. Basically the second url: http://localhost:62737/browser_testing/rt/InBrowserTesting.js would really need to look like: http://ctier.arcode.com:8888/ <http://ctier.arcode.com:8888/foo.html>browser_testing/rt/InBrowserTesting.js <http://localhost:62737/browser_testing/rt/InBrowserTesting.js> regards, Samuele Pedroni From lac at openend.se Fri Apr 9 01:14:03 2010 From: lac at openend.se (Laura Creighton) Date: Fri, 09 Apr 2010 01:14:03 +0200 Subject: [py-dev] oesjkit In-Reply-To: Message from Samuele Pedroni <pedronis@openend.se> of "Thu, 08 Apr 2010 13:47:33 +0200." <4BBDC255.3020206@openend.se> References: <TGmx3i_RRTv0XKAG6Ejdc-2IGXlQdAvMVKkYvDbAjhzwtsJFc@smtp.gmail.com> <4BBDC255.3020206@openend.se> Message-ID: <201004082314.o38NE3aN010030@theraft.openend.se> Any idea how to get it listed here? http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy#MockTestingTools Laura (still ill, but reading some mail now at any rate) From phil at freehackers.org Mon Apr 12 12:24:12 2010 From: phil at freehackers.org (Philippe Fremy) Date: Mon, 12 Apr 2010 12:24:12 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level In-Reply-To: <20091104172342.GU29141@trillke.net> References: <49F6EA81.2070106@freehackers.org> <20090428115556.GJ11963@trillke.net> <49F71C0E.3000800@freehackers.org> <20090428180200.GO11963@trillke.net> <4AE80A42.5030009@freehackers.org> <20091104172342.GU29141@trillke.net> Message-ID: <4BC2F4CC.9040408@freehackers.org> An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20100412/5307d5db/attachment.html> From phil at freehackers.org Mon Apr 12 12:37:36 2010 From: phil at freehackers.org (Philippe Fremy) Date: Mon, 12 Apr 2010 12:37:36 +0200 Subject: [py-dev] order of setup / teardown Message-ID: <4BC2F7F0.1030603@freehackers.org> Hi, I am not sure if this has been brought before, but I am surprised to see some calls inside the test "hooks" after the test session has finished. I am referring here to teardown_class, revealed by a quick example: pytest_sessionstart setup_class setup_method test1 teardown_method setup_method test2 teardown_method pytest_sessionfinish teardown_class This is creating problems for me because I am assuming that all resources are setup in sesssionstart, and torn down in sessionfinish. And teardown_class is using one of those resources. Is it a bug or intended behavior ? cheers, Philippe From holger at merlinux.eu Mon Apr 12 13:13:08 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 12 Apr 2010 13:13:08 +0200 Subject: [py-dev] order of setup / teardown In-Reply-To: <4BC2F7F0.1030603@freehackers.org> References: <4BC2F7F0.1030603@freehackers.org> Message-ID: <20100412111308.GY26514@trillke.net> Hi Philippe, (note to all: i am back from a long and fun vacation and will see to answer pending issues in the next days) On Mon, Apr 12, 2010 at 12:37 +0200, Philippe Fremy wrote: > Hi, > > I am not sure if this has been brought before, but I am surprised to see > some calls inside the test "hooks" after the test session has finished. > > I am referring here to teardown_class, revealed by a quick example: > > pytest_sessionstart > > setup_class > setup_method > > test1 > > teardown_method > > setup_method > test2 > > teardown_method > > pytest_sessionfinish > > teardown_class > > This is creating problems for me because > I am assuming that all resources are setup in sesssionstart, > and torn down in sessionfinish. And teardown_class is using one of > those resources. > > Is it a bug or intended behavior ? setup and teardown of python test functions are implemented through the builtin py/_plugin/pytest_runner.py plugin. It itself implements hooks called from the "session" in order to provide the setup/teardown semantics. There, it only tears down class/module state if the "next" test does not need it and this is decided at setup time of said next test. This means that after the last test has run, some resources might still require teardown and currently this is done through implementing it in a pytest_sessionfinish hook. The latter, however, is invoked after all user-provided hooks have been invoked. So much for the analysis. Also related to your other "directory-setup" issue i am not entirely happy about pointing people to pytest_namespace and sessionstart/sessionfinish for scoped python setup purposes. Maybe more python-specific setup/teardown hook(s) would make more sense? Relatedly, I ponder giving up on the incremental collection of tests and internally immediately produce a fully exploded lists of tests at session start. This way teardown could be done in a more precise manner -- right after the resource/scope is not needed anymore. It would also make randomization and other test re-ordering easy. A downside is that it takes longer for the first test to run. A plus is that it probably eases implementation and provides less surprises (such as the one you are having because pytest_runner would not need to work through sessionfinish anymore). If you don't tell anyone - :) - there also is a immediate work around to your issue i think. You could implement something like def pytest_sessionfinish(__multicall__): __multicall__.execute() # call other sessionfinish implementers first '__multicall__' is the object representing the current ongoing plugin hook call and allows to dynamically call other reamining hook implementers first. It's not documented or advertised (but used sometimes from pytest's own builtin plugins) because i think a more declarative decorator-based way of setting call ordering is saner. Then again, i hope to avoid the need altogether for simplicity sake. cheers, holger From Ronny.Pfannschmidt at gmx.de Tue Apr 13 14:19:11 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 13 Apr 2010 14:19:11 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level In-Reply-To: <4BC2F4CC.9040408@freehackers.org> References: <49F6EA81.2070106@freehackers.org> <20090428115556.GJ11963@trillke.net> <49F71C0E.3000800@freehackers.org> <20090428180200.GO11963@trillke.net> <4AE80A42.5030009@freehackers.org> <20091104172342.GU29141@trillke.net> <4BC2F4CC.9040408@freehackers.org> Message-ID: <1271161151.9772.4.camel@localhost> On Mon, 2010-04-12 at 12:24 +0200, Philippe Fremy wrote: > > That thread is 6 months old, but I see that it is still relevant. > > holger krekel wrote: > > Hi Philippe, > > > > On Wed, Oct 28, 2009 at 10:09 +0100, Philippe Fremy wrote: > > > > > I had a quick look at the plugin architecture to see if I could > > > implement an equivalent of setup/teardown at directory level, but I > > > don't think it's possible. > > > > > > > Whatever is called for "directory" setup could live in a conftest.py file. > > The question is how to transfer any "directory" setup state > > to a module. At first i thought one could write down: > > > > # ./conftest.py > > def setup_module(mod): > > # will be called for all modules in/below the dir > > mod.something = "value" > > > > but one would expect this to be called for each module and not > > just once for a whole directory. Now one could perform some > > directory-level caching but one conceptual issue remains: > > values would be "magically" injected into the test modules. > > > > Do you have ideas about how you'd like the API to work? > > > > Yes, I now know how I want to transfer the value. I would like to use > the py.test namespace. > > Today, only pytest_namespace() is available to setup an initial > namespace. I would like to see a pytest_update_namespace() to pass a > dict with key/values to put in the namespace. > > cheers, > > Philippe > How about more control over funcargs in particular a) caching on various levels of packages b) exposure of all currently usable funcargs in the py.test namespace -- Ronny From memedough at gmail.com Fri Apr 16 11:09:12 2010 From: memedough at gmail.com (meme dough) Date: Fri, 16 Apr 2010 19:09:12 +1000 Subject: [py-dev] xdist sending info between master and slave - support for client code Message-ID: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> Hi, I'm using pytest and the xdist plugin. I'd like to get some information from the master to the slave, specifically on the slave side I need to know the master's host, master's topdir and master's rsync roots. Currently I modified xdist to send the required info over the channel in install_slave: master_host = socket.gethostname() master_topdir = config.topdir channel.send((config, basetemp, gateway.id, master_host, master_topdir, roots)) The rsync roots were passed into the TXNode during creation by the NodeManager: node = TXNode(gateway, self.config, putevent, self.roots) That gets the info there but I'm not sure of the best way. The master host / topdir could be added to config.options since that get pickled over the channel, but it's not a command line option so it doesn't seem the best place to put that info. But it would work from a client point of view without modifying xdist. The master rsync roots can't be worked out by client unless it reimplements the NodeManager._getrsyncdirs logic. This happens because DSession does setup which creates NodeManager and setup_nodes which does install_slave without any client hook in between so at no point can client get rsync roots from session.nodemanager.roots and put them in something like config.options to get to slave side. That is the first half. Next I need to send some info back from slave to master when slave has finished. Currently I modified xdist to send back a report object when the slave has finished. The SlaveNode has a report object which any client hook can change or add to if they want to send info back to master: self.report = None I use pytest_sessionfinish hook on slave to set report to dict with info. When slave finish it sendevent with the report: self.sendevent("slavefinished", report=self.report) The TXNode on master gets report and stores the slave report on itself: self.report = kwargs['report'] Then on master use pytest_testnodedown hook to get slave report off TXNode. That gets the info back from slave to master but I'm not sure if that's best way. Is it possible to modify xdist so that client code on master can send info to the slave. Will need access to rsync roots before it goes ahead and installs the slave though. Is it possible to modify xdist so that client code on the slave can send info back to the master. This looks like must modify xdist somehow since it doesn't seem to send any report back at finish. Now that I wrote all that, what is the obvious answer that I didn't see? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20100416/0579940b/attachment.html> From holger at merlinux.eu Fri Apr 16 19:21:44 2010 From: holger at merlinux.eu (holger krekel) Date: Fri, 16 Apr 2010 19:21:44 +0200 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> Message-ID: <20100416172143.GY26514@trillke.net> Hi meme (correct addressing? :) thanks for the mail. For my understanding, can you mention a bit more context? On Fri, Apr 16, 2010 at 19:09 +1000, meme dough wrote: [...] > Is it possible to modify xdist so that client code on master can send info > to the slave. Will need access to rsync roots before it goes ahead and > installs the slave though. I think we could add a hook called at the master-side which could return/update an extra dictionary to be transfered to the client (in theory even a pickled-object but i'd like to avoid guaranteeing pickleability for the core test distribution mechanism). A slave-side hook (sessionstart?) would get this information. > Is it possible to modify xdist so that client code on the slave can send > info back to the master. This looks like must modify xdist somehow since it > doesn't seem to send any report back at finish. We can add a hook or re-use an existing one for allowing a client to send back (e.g. report) data but the question is: which hook sees this information on the master side? Maybe sessionfinish? So the slave's sessionfinish hook could add whatever extra information and the final sessionfinish sees it. Maybe it's better to design and call specific xdist-prefied hooks and not re-use existing ones? As you have hacked on this topic and for your purposes already - what do you think? > Now that I wrote all that, what is the obvious answer that I didn't see? > > :) I don't have an obvious answer but then again after a long holiday i still need to dive back a bit more; maybe i find some then :) cheers, holger From holger at merlinux.eu Fri Apr 16 19:45:07 2010 From: holger at merlinux.eu (holger krekel) Date: Fri, 16 Apr 2010 19:45:07 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level In-Reply-To: <1271161151.9772.4.camel@localhost> References: <49F6EA81.2070106@freehackers.org> <20090428115556.GJ11963@trillke.net> <49F71C0E.3000800@freehackers.org> <20090428180200.GO11963@trillke.net> <4AE80A42.5030009@freehackers.org> <20091104172342.GU29141@trillke.net> <4BC2F4CC.9040408@freehackers.org> <1271161151.9772.4.camel@localhost> Message-ID: <20100416174507.GZ26514@trillke.net> Hi Philippe, Ronny, On Tue, Apr 13, 2010 at 14:19 +0200, Ronny Pfannschmidt wrote: > On Mon, 2010-04-12 at 12:24 +0200, Philippe Fremy wrote: > > > > That thread is 6 months old, but I see that it is still relevant. > > > > holger krekel wrote: > > > Hi Philippe, > > > > > > On Wed, Oct 28, 2009 at 10:09 +0100, Philippe Fremy wrote: > > > > > > > I had a quick look at the plugin architecture to see if I could > > > > implement an equivalent of setup/teardown at directory level, but I > > > > don't think it's possible. > > > > > > > > > > Whatever is called for "directory" setup could live in a conftest.py file. > > > The question is how to transfer any "directory" setup state > > > to a module. At first i thought one could write down: > > > > > > # ./conftest.py > > > def setup_module(mod): > > > # will be called for all modules in/below the dir > > > mod.something = "value" > > > > > > but one would expect this to be called for each module and not > > > just once for a whole directory. Now one could perform some > > > directory-level caching but one conceptual issue remains: > > > values would be "magically" injected into the test modules. > > > > > > Do you have ideas about how you'd like the API to work? > > > > > > > Yes, I now know how I want to transfer the value. I would like to use > > the py.test namespace. > > > > Today, only pytest_namespace() is available to setup an initial > > namespace. I would like to see a pytest_update_namespace() to pass a > > dict with key/values to put in the namespace. > > > > cheers, > > > > Philippe > > > > How about more control over funcargs > > in particular > > a) caching on various levels of packages > b) exposure of all currently usable funcargs in the py.test namespace Well, funcargs are well defined when used/requested from a test function. Having them globally available blurs the lines a lot and convolutes the py.test namespace. and what would e.g. py.test.tmpdir even be? would it be different when called from a test func, the various setup functions or used at import time etc.? Getting back to Philippe's goal of having per-directory cached values exposed to test functions i wonder if we could a) introduce a directory scope for funcarg-caching b) allow (a subset of) funcargs for setup functions Given this we could write down something like: # ./test_module.py def setup_module(mod, appserver): mod.appserver = appserver # ./conftest.py def pytest_funcarg__appserver(request): return request.cached_setup( setup=lambda: MyAppServer(...), teardown=lambda x: x.shutdown(), scope="directory" ) However, i am not sure if allowing funcargs in setup functions is sane. Also i am not thrilled about encouraging the use of global variables for transfering information. Maybe implementing a "directory" scope for cached_setup would go a long way. Any test function could then directly use the 'appserver' object as a function argument and rely on its per-directory creation/teardown. best, holger From memedough at gmail.com Sat Apr 17 05:54:13 2010 From: memedough at gmail.com (meme dough) Date: Sat, 17 Apr 2010 13:54:13 +1000 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <20100416172143.GY26514@trillke.net> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> Message-ID: <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> Hi, > Is it possible to modify xdist so that client code on master can send info > > > to the slave. ?Will need access to rsync roots before it goes ahead and > > installs the slave though. > > I think we could add a hook called at the master-side which could return/update > an extra dictionary to be transfered to the client (in theory even a > pickled-object but i'd like to avoid guaranteeing pickleability for the core > test distribution mechanism). ?A slave-side hook (sessionstart?) would > get this information. I created a new hook that gets called just before each test node is installed: def pytest_testnodeinstall(nodemanager, gateway, slavedata): Client code on the master can set anything it needs to send to the slave on the slavedata object. The slavedata is passed over the channel in the same way as config object and is stored on the slave node such that any client code on the slave can access the slavedata. It can do this in any hook that the slave client code needs to implement (provided that it can get access to the session object which is the slave node object which has the slavedata). > > Is it possible to modify xdist so that client code on the slave can send > > info back to the master. ?This looks like must modify xdist somehow since it > > doesn't seem to send any report back at finish. > > We can add a hook or re-use an existing one for allowing a client to send > back (e.g. report) data but the question is: which hook sees this information > on the master side? > > Maybe sessionfinish? ?So the slave's sessionfinish hook could add whatever > extra information and the final sessionfinish sees it. When the slave node is created it sets a slavereport to an empty slavereport object. Client code on the slave can add to the slavereport in any hook that it implements. When the slave finishes it already sends a slavefinished event which now also sends back the slavereport object. The master TXNode stores the slavereport on itself. Client code on the master can implement the pytest_testnodedown hook and access the slavereport. > Maybe it's better to design and call specific xdist-prefied hooks and not re-use > existing ones? ?As you have hacked on this topic and for your purposes > already - what do you think? xdist hooks seem better than the existing ones. Also many slaves send reports back on pytest_testnodedown hook and seems better to deal with just the one immediately rather than build up and pass to pytest_sessionfinish. I implemented and if you think it is good enough would you please be so kind as to pull the changes in and release pytest and xdist. You can change of course if you have a better idea! http://bitbucket.org/memedough/py-trunk/overview http://bitbucket.org/memedough/pytest-xdist/overview :) From holger at merlinux.eu Mon Apr 19 13:04:54 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 19 Apr 2010 13:04:54 +0200 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> Message-ID: <20100419110454.GC26514@trillke.net> On Sat, Apr 17, 2010 at 13:54 +1000, meme dough wrote: > > Is it possible to modify xdist so that client code on master can send info > > > > > to the slave. ?Will need access to rsync roots before it goes ahead and > > > installs the slave though. > > > > I think we could add a hook called at the master-side which could return/update > > an extra dictionary to be transfered to the client (in theory even a > > pickled-object but i'd like to avoid guaranteeing pickleability for the core > > test distribution mechanism). ?A slave-side hook (sessionstart?) would > > get this information. > > I created a new hook that gets called just before each test node is installed: > > def pytest_testnodeinstall(nodemanager, gateway, slavedata): I think we can just reuse testnodeready and carry a "nodemanager" and a slavedata attribute on each node. Sidenote: at some point i'd like to change the distribution model and thus would like to hide as many xdist-internals as possible for now. The goal for that change is to do test collection on the clients and co-ordinate only their execution and reporting from the master. > Client code on the master can set anything it needs to send to the > slave on the slavedata object. > > The slavedata is passed over the channel in the same way as config > object and is stored on the slave node such that any client code on > the slave can access the slavedata. > It can do this in any hook that > the slave client code needs to implement (provided that it can get > access to the session object which is the slave node object which has > the slavedata). I'd like slavedata to be a simple dictionary so that slave code can access it by "session.slavedata['NAME']". Btw, I actually consider it a bit of a hack that 'session == node' on slaves. But apparently useful :) > > > Is it possible to modify xdist so that client code on the slave can send > > > info back to the master. ?This looks like must modify xdist somehow since it > > > doesn't seem to send any report back at finish. > > > > We can add a hook or re-use an existing one for allowing a client to send > > back (e.g. report) data but the question is: which hook sees this information > > on the master side? > > > > Maybe sessionfinish? ?So the slave's sessionfinish hook could add whatever > > extra information and the final sessionfinish sees it. > > When the slave node is created it sets a slavereport to an empty > slavereport object. > > Client code on the slave can add to the slavereport in any hook that > it implements. only in hooks that know about 'session' or nodes respectively, right? I wonder if we could settle on a one-to-one config<->node relationship on the slave side. Would make the node nicely available. > When the slave finishes it already sends a slavefinished event which > now also sends back the slavereport object. > > The master TXNode stores the slavereport on itself. > > Client code on the master can implement the pytest_testnodedown hook > and access the slavereport. sounds ok enough. > > Maybe it's better to design and call specific xdist-prefied hooks and not re-use > > existing ones? ?As you have hacked on this topic and for your purposes > > already - what do you think? > > xdist hooks seem better than the existing ones. Also many slaves send > reports back on pytest_testnodedown hook and seems better to deal with > just the one immediately rather than build up and pass to > pytest_sessionfinish. Can you finally briefly describe your use case - i.e. for what are you using the new master<->slave data exchange? > I implemented and if you think it is good enough would you please be > so kind as to pull the changes in and release pytest and xdist. You > can change of course if you have a better idea! > > http://bitbucket.org/memedough/py-trunk/overview > > http://bitbucket.org/memedough/pytest-xdist/overview Could you see to implement my above suggestions and see if it still satisfies your use case? (reuse testnodeready, slavedata dict, config<->node relationship on slaves) If i see it right that'd confine changes to the xdist code in which case i'd be willing to release it soon. I'd be happy if you could also add tests for the new functionality. cheers & thanks for the patch and explanations, holger > :) > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- From prologic at shortcircuit.net.au Wed Apr 21 04:43:44 2010 From: prologic at shortcircuit.net.au (James Mills) Date: Wed, 21 Apr 2010 12:43:44 +1000 Subject: [py-dev] Testing and Coverage against different python interpreter versions Message-ID: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> Hi All, I am in need of a solution regarding the use of py.test and pytest-coverage (the coverage plugin I help maintain for py.test). The problem is being able to run a test suite against 2 or more different python interpreter versions (eg: 2.5 and 2.6) AND combining their coverage data. Without the above behavior, it would not be possible to achieve 100% code coverage (by tests). Often (at least I do) in situations where you wish to support 2 or more different versions of python you might have: try: import foo # 2.6 import except ImportError: import bar # 2.5 import and if sys.version_info[:2] == (2, 6): # 2.6 code else: # 2.5 code Currently I have no solution for this, I'm hoping others might have some ideas or are already working on something similar that might achieve this. cheers James -- -- "Problems are solved by method" From memedough at gmail.com Wed Apr 21 10:15:51 2010 From: memedough at gmail.com (meme dough) Date: Wed, 21 Apr 2010 18:15:51 +1000 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> Message-ID: <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> Hi, I have written a new pytest coverage plugin from the ground up to fully support distributed testing since both pytest and coverage provide good support for doing this. Plus it gives access to pretty much everything that coverage has that I know about. The use case that you state is interesting. Setting dist=load will cause pytest to spread the tests out to many slaves with a single coverage report from their combined data. However each node is running different tests so setting them to different python / platform may not give full coverage. Setting dist=each will cause pytest to gives all tests to each slave with a report from each slave. However each report will just be for the one slave. I had not thought of the case you suggest where it may be desirable to instruct the plugin to actually combine the results from each slave where each slave has run all tests. I will consider this further. :) On 21 April 2010 12:43, James Mills <prologic at shortcircuit.net.au> wrote: > Hi All, > > I am in need of a solution regarding the use of py.test and pytest-coverage > (the coverage plugin I help maintain for py.test). > > The problem is being able to run a test suite against 2 or more different > python interpreter versions (eg: 2.5 and 2.6) AND combining their > coverage data. > > Without the above behavior, it would not be possible to achieve 100% code > coverage (by tests). Often (at least I do) in situations where you > wish to support > 2 or more different versions of python you might have: > > try: > ? import foo # 2.6 import > except ImportError: > ? import bar # 2.5 import > > and > > if sys.version_info[:2] == (2, 6): > ? # 2.6 code > else: > ? # 2.5 code > > Currently I have no solution for this, I'm hoping others might have > some ideas or are already working on something similar that might > achieve this. > > cheers > James > > -- > -- "Problems are solved by method" > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From prologic at shortcircuit.net.au Wed Apr 21 10:23:37 2010 From: prologic at shortcircuit.net.au (James Mills) Date: Wed, 21 Apr 2010 18:23:37 +1000 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> Message-ID: <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> On Wed, Apr 21, 2010 at 6:15 PM, meme dough <memedough at gmail.com> wrote: > I have written a new pytest coverage plugin from the ground up to > fully support distributed testing since both pytest and coverage > provide good support for doing this. ?Plus it gives access to pretty > much everything that coverage has that I know about. Do you have a link to this at all ? Do you intend to maintain and support this new plugin ? I'm happy to drop the one I maintain on bitbucket in favor of yours if need be. > I had not thought of the case you suggest where it may be desirable to > instruct the plugin to actually combine the results from each slave > where each slave has run all tests. > > I will consider this further. Please do as this is an important use-case ihmo (and others agree). (Maybe not for distributed testing, but certainly for testing a library against more than one supported version of python). Look forward to using your work, cheers James From holger at merlinux.eu Wed Apr 21 10:34:02 2010 From: holger at merlinux.eu (holger krekel) Date: Wed, 21 Apr 2010 10:34:02 +0200 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> Message-ID: <20100421083402.GG26514@trillke.net> On Wed, Apr 21, 2010 at 18:23 +1000, James Mills wrote: > On Wed, Apr 21, 2010 at 6:15 PM, meme dough <memedough at gmail.com> wrote: > > I have written a new pytest coverage plugin from the ground up to > > fully support distributed testing since both pytest and coverage > > provide good support for doing this. ?Plus it gives access to pretty > > much everything that coverage has that I know about. > > Do you have a link to this at all ? > Do you intend to maintain and support this new plugin ? > > I'm happy to drop the one I maintain on bitbucket in favor of yours if need be. Let's discuss and review things when they are there. > > I had not thought of the case you suggest where it may be desirable to > > instruct the plugin to actually combine the results from each slave > > where each slave has run all tests. > > > > I will consider this further. > > Please do as this is an important use-case ihmo (and others agree). I often run tests on multiple platforms and i'd appreciate a combined coverage report. Moreover, i'd like to see how to then only report skips when they were skipped on all platforms. This way it'd be easy to identify which platform/interpreter is missing in the testing mix by looking at remaining skips. cheers, holger > (Maybe not for distributed testing, but certainly for testing a library > against more than one supported version of python). > > Look forward to using your work, > > cheers > James > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -- From memedough at gmail.com Wed Apr 21 14:47:25 2010 From: memedough at gmail.com (meme dough) Date: Wed, 21 Apr 2010 22:47:25 +1000 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <20100419110454.GC26514@trillke.net> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> <20100419110454.GC26514@trillke.net> Message-ID: <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> Hi, > Could you see to implement my above suggestions and see > if it still satisfies your use case? > (reuse testnodeready, slavedata dict, > config<->node relationship on slaves) > > If i see it right that'd confine changes to the xdist code > in which case i'd be willing to release it soon. I changed to use slaveready and changed both the slavedata and slavereport to be a simple dict. Not sure with the relationships on slaves. The hooks just access the session / slave object to get config / slavedata / slavereport. Seems clear to me atm, but maybe I misunderstood. I added test which works, but could maybe be better when it fails. Wasn't sure how to best make a test of pytest with a hook running on a slave fail. What do you think? http://bitbucket.org/memedough/pytest-xdist-slaveready/overview :) From holger at merlinux.eu Wed Apr 21 21:27:24 2010 From: holger at merlinux.eu (holger krekel) Date: Wed, 21 Apr 2010 21:27:24 +0200 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> <20100419110454.GC26514@trillke.net> <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> Message-ID: <20100421192724.GI26514@trillke.net> Hi Meme, On Wed, Apr 21, 2010 at 22:47 +1000, meme dough wrote: > > Could you see to implement my above suggestions and see > > if it still satisfies your use case? > > (reuse testnodeready, slavedata dict, > > config<->node relationship on slaves) > > > > If i see it right that'd confine changes to the xdist code > > in which case i'd be willing to release it soon. > > I changed to use slaveready and changed both the slavedata and > slavereport to be a simple dict. thanks. > Not sure with the relationships on slaves. The hooks just access the > session / slave object to get config / slavedata / slavereport. Seems > clear to me atm, but maybe I misunderstood. Only sessionstart/sessionfinish get a 'session' (node) object while all other hooks can not easily obtain a session reference. However, almost all hooks receive or can easily get to a 'config' object. Therefore i think it's better to stick the slavedata object to the config object. > I added test which works, but could maybe be better when it fails. > Wasn't sure how to best make a test of pytest with a hook running on a > slave fail. > > What do you think? > > http://bitbucket.org/memedough/pytest-xdist-slaveready/overview The test looks fine. Looking at your patch and at things some more i wonder if it isn't eventually better to introduce a new hook - even if it requires a new py.test release. Something like: # runs only on the master side def pytest_xdist_configure_node(node): node.slaveinput[NAME] = value # any hook or test can do the following manipulation if hasattr(config, 'slavedata'): # means we are running slave side config.slaveinput[NAME] # access data sent from the master config.slaveoutput['result'] = 42 # set slave output # runs only on the master side def pytest_xdist_nodedown(node, slaveoutput): node.slaveoutput['result'] # access result from slave I think both internal implementation and usage becomes simpler. What do you think? When doing the a new py.test release i'd like to look into allowing to add per-plugin hooks and systematically name xdist-hooks. If you'd like to push ahead before i finish this you can just add the hooks statically and i patch it out later. cheers, holger From holger at merlinux.eu Thu Apr 22 12:05:25 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 22 Apr 2010 12:05:25 +0200 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <20100421192724.GI26514@trillke.net> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> <20100419110454.GC26514@trillke.net> <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> <20100421192724.GI26514@trillke.net> Message-ID: <20100422100525.GM26514@trillke.net> On Wed, Apr 21, 2010 at 21:27 +0200, holger krekel wrote: > When doing the a new py.test release i'd like to look into > allowing to add per-plugin hooks and systematically name xdist-hooks. Just did checkins to py-trunk and xdist implementing a first go at this, bumping internal version numbers to 1.2.2 and 1.2 respectively. (hudson testruns are ongoing at http://hudson.testrun.org/view/pytest/) > If you'd like to push ahead before i finish this you can just add the > hooks statically and i patch it out later. It should be easy to add new hooks now to xdist/newhooks.py holger From memedough at gmail.com Thu Apr 22 14:35:12 2010 From: memedough at gmail.com (meme dough) Date: Thu, 22 Apr 2010 22:35:12 +1000 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <20100422100525.GM26514@trillke.net> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> <20100419110454.GC26514@trillke.net> <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> <20100421192724.GI26514@trillke.net> <20100422100525.GM26514@trillke.net> Message-ID: <y2pcf54a0ac1004220535obe97edabh7c5593b936d5b924@mail.gmail.com> Hi, > Just did checkins to py-trunk and xdist implementing a first go at this, > bumping internal version numbers to 1.2.2 and 1.2 respectively. > (hudson testruns are ongoing at http://hudson.testrun.org/view/pytest/) > >> If you'd like to push ahead before i finish this you can just add the >> hooks statically and i patch it out later. > > It should be easy to add new hooks now to xdist/newhooks.py I've pulled in changes and looks good. Another commit to xdist for putting slaveinput / slaveoutput on the config object for slaves. For master remains on the node since many nodes, this also allows checking if running as slave as you pointed out. I renamed to slaveinput / slaveoutput as had in previous email. I also changed the test so that instead of asserts in the hooks on slave or master it will output something at summary and the test process can check. http://bitbucket.org/memedough/pytest-xdist-slaveready-config/changeset/5257145dbd4b An issue though: Last execnet commit is 1.0.6 version bump but not tagged or released on pypi. Hence can't install or develop pytest-xdist. I changed locally but didn't put in this changeset. Not sure if you want to down xdist to point 1.0.5 or release execnet 1.0.6. :) From memedough at gmail.com Thu Apr 22 15:50:57 2010 From: memedough at gmail.com (meme dough) Date: Thu, 22 Apr 2010 23:50:57 +1000 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <20100421192724.GI26514@trillke.net> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> <20100419110454.GC26514@trillke.net> <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> <20100421192724.GI26514@trillke.net> Message-ID: <t2zcf54a0ac1004220650kdb7c7eb5pfb5a912449259685@mail.gmail.com> Hi, > The test looks fine. ?Looking at your patch and at things some more i > wonder if it isn't eventually better to introduce a new hook - even > if it requires a new py.test release. ?Something like: > > ? ?# runs only on the master side > ? ?def pytest_xdist_configure_node(node): > ? ? ? ?node.slaveinput[NAME] = value > > ? ?# any hook or test can do the following manipulation > ? ? ? ?if hasattr(config, 'slavedata'): # means we are running slave side > ? ? ? ? ? ?config.slaveinput[NAME] ? ? ?# access data sent from the master > ? ? ? ? ? ?config.slaveoutput['result'] = 42 ?# set slave output > > ? ?# runs only on the master side > ? ?def pytest_xdist_nodedown(node, slaveoutput): > ? ? ? ?node.slaveoutput['result'] # access result from slave > > I think both internal implementation and usage becomes simpler. > What do you think? It looks good. The hook names can be made consistent as well as consistent args. :) From holger at merlinux.eu Thu Apr 22 17:03:26 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 22 Apr 2010 17:03:26 +0200 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <y2pcf54a0ac1004220535obe97edabh7c5593b936d5b924@mail.gmail.com> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> <20100419110454.GC26514@trillke.net> <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> <20100421192724.GI26514@trillke.net> <20100422100525.GM26514@trillke.net> <y2pcf54a0ac1004220535obe97edabh7c5593b936d5b924@mail.gmail.com> Message-ID: <20100422150326.GA22675@trillke.net> Hi Meme, On Thu, Apr 22, 2010 at 22:35 +1000, meme dough wrote: > > Just did checkins to py-trunk and xdist implementing a first go at this, > > bumping internal version numbers to 1.2.2 and 1.2 respectively. > > (hudson testruns are ongoing at http://hudson.testrun.org/view/pytest/) > > > >> If you'd like to push ahead before i finish this you can just add the > >> hooks statically and i patch it out later. > > > > It should be easy to add new hooks now to xdist/newhooks.py > > I've pulled in changes and looks good. > > Another commit to xdist for putting slaveinput / slaveoutput on the > config object for slaves. For master remains on the node since many > nodes, this also allows checking if running as slave as you pointed > out. > > I renamed to slaveinput / slaveoutput as had in previous email. > > I also changed the test so that instead of asserts in the hooks on > slave or master it will output something at summary and the test > process can check. > > http://bitbucket.org/memedough/pytest-xdist-slaveready-config/changeset/5257145dbd4b I pulled it and did another patch to use a new configure-node hook. > An issue though: > > Last execnet commit is 1.0.6 version bump but not tagged or released > on pypi. Hence can't install or develop pytest-xdist. I changed > locally but didn't put in this changeset. Not sure if you want to > down xdist to point 1.0.5 or release execnet 1.0.6. Both - xdist now only requires 1.0.5 but i'd like to put out 1.0.6 soonish. A question - do i guess right that your distributed coverage plugin is to use the new slaveinput/slaveoutput mechanism? cheers, holger > :) > -- From memedough at gmail.com Sat Apr 24 03:20:41 2010 From: memedough at gmail.com (meme dough) Date: Sat, 24 Apr 2010 11:20:41 +1000 Subject: [py-dev] xdist sending info between master and slave - support for client code In-Reply-To: <20100422150326.GA22675@trillke.net> References: <g2jcf54a0ac1004160209x3aebf4bak1c3acac2a3ba283d@mail.gmail.com> <20100416172143.GY26514@trillke.net> <w2wcf54a0ac1004162054jc0fd38a2zdc8f8f2b41f70928@mail.gmail.com> <20100419110454.GC26514@trillke.net> <m2kcf54a0ac1004210547pdcee4817ybd9e25fe96af09ee@mail.gmail.com> <20100421192724.GI26514@trillke.net> <20100422100525.GM26514@trillke.net> <y2pcf54a0ac1004220535obe97edabh7c5593b936d5b924@mail.gmail.com> <20100422150326.GA22675@trillke.net> Message-ID: <o2zcf54a0ac1004231820u475049es80ba87073367b19e@mail.gmail.com> Hi, >> > Just did checkins to py-trunk and xdist implementing a first go at this, >> > bumping internal version numbers to 1.2.2 and 1.2 respectively. >> > (hudson testruns are ongoing at http://hudson.testrun.org/view/pytest/) >> > >> >> If you'd like to push ahead before i finish this you can just add the >> >> hooks statically and i patch it out later. >> > >> > It should be easy to add new hooks now to xdist/newhooks.py >> >> I've pulled in changes and looks good. >> >> Another commit to xdist for putting slaveinput / slaveoutput on the >> config object for slaves. ?For master remains on the node since many >> nodes, this also allows checking if running as slave as you pointed >> out. >> >> I renamed to slaveinput / slaveoutput as had in previous email. >> >> I also changed the test so that instead of asserts in the hooks on >> slave or master it will output something at summary and the test >> process can check. >> >> http://bitbucket.org/memedough/pytest-xdist-slaveready-config/changeset/5257145dbd4b > > I pulled it and did another patch to use a new configure-node hook. This new hook is better. I didn't like the second receive to get the slaveinput and having to check on each send. More clear to send slaveinput over at same time as config. >> An issue though: >> >> Last execnet commit is 1.0.6 version bump but not tagged or released >> on pypi. ?Hence can't install or develop pytest-xdist. ?I changed >> locally but didn't put in this changeset. ?Not sure if you want to >> down xdist to point 1.0.5 or release execnet 1.0.6. > > Both - xdist now only requires 1.0.5 but i'd like to put out 1.0.6 soonish. > > A question - do i guess right that your distributed coverage plugin > is to use the new slaveinput/slaveoutput mechanism? Yes, I will move it to the latest xdist now. Thanks for the release, this is really good. :) From memedough at gmail.com Sat Apr 24 16:01:02 2010 From: memedough at gmail.com (meme dough) Date: Sun, 25 Apr 2010 00:01:02 +1000 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> Message-ID: <v2gcf54a0ac1004240701u4dfc94a1pcc3a5f160bb8828@mail.gmail.com> Hi, >> I have written a new pytest coverage plugin from the ground up to >> fully support distributed testing since both pytest and coverage >> provide good support for doing this. ?Plus it gives access to pretty >> much everything that coverage has that I know about. > > Do you have a link to this at all ? > Do you intend to maintain and support this new plugin ? With the latest py and pytest-xdist release the master and slave can exchange information. Until this release I had local mods and suggested patches. Thanks Holger! There are still a number of loose ends to tidy up but it's now up on pypi and bitbucket so you can throw it into a virtualenv and take it for a test run if you like. http://pypi.python.org/pypi/pytest-cover/ http://bitbucket.org/memedough/pytest-cover I will definitely be using and maintaining it. > I'm happy to drop the one I maintain on bitbucket in favor of yours if need be. I didn't know quite what to do about the pytest-coverage repo. I thought of contributing to it, however it's a complete rewrite so it's a bit much to ask to take a patch that probably only has pytest hook lines in common. Then I also kind of didn't like the repo having the figleaf history and the 1.0 tag when that latest tag was only at 0.3. Maybe I'm just being picky and shouldn't... Not sure it should be dropped though, it's there, stable, and probably being used by people. And since I didn't patch the existing one but wrote something new it's currently unproven. Maybe best to see how things stack up after a while, doesn't seem any harm having options - we have figleaf too. Lets just wait for a while - maybe we merge some things and get best of all. >> I had not thought of the case you suggest where it may be desirable to >> instruct the plugin to actually combine the results from each slave >> where each slave has run all tests. >> >> I will consider this further. > > Please do as this is an important use-case ihmo (and others agree). > (Maybe not for distributed testing, but certainly for testing a library > against more than one supported version of python). This is a good suggestion, I added it so you can do dist=each and use --cover-combine-each option to get single combined report from all slaves. Do note however that I haven't tested on all python versions so I probably have some work to do. Should be alright if you're only doing 2.5 / 2.6 as your email suggested though. :) From memedough at gmail.com Sun Apr 25 04:06:00 2010 From: memedough at gmail.com (meme dough) Date: Sun, 25 Apr 2010 12:06:00 +1000 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <v2gcf54a0ac1004240701u4dfc94a1pcc3a5f160bb8828@mail.gmail.com> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> <v2gcf54a0ac1004240701u4dfc94a1pcc3a5f160bb8828@mail.gmail.com> Message-ID: <u2ycf54a0ac1004241906t57d5f488x87d8a421c4b446db@mail.gmail.com> Hi, > http://pypi.python.org/pypi/pytest-cover/ > > http://bitbucket.org/memedough/pytest-cover Renamed and moved to: http://pypi.python.org/pypi/pytest-cov http://bitbucket.org/memedough/pytest-cov It now coexists nicely with other plugins. :) From memedough at gmail.com Mon Apr 26 06:49:46 2010 From: memedough at gmail.com (meme dough) Date: Mon, 26 Apr 2010 14:49:46 +1000 Subject: [py-dev] distributed test slave activation of external plugin Message-ID: <v2rcf54a0ac1004252149x7af83806te1d3ce921ff7599e@mail.gmail.com> Hi, Is there any way to activate external plugins for distributed testing on the slave side? Currently pytest-cov requires that it is installed on the slave version of python. This differs from pytest-xdist that will rsync itself across, such that the only requirement on the slave side is python. If pytest-cov is not installed on the slave then the master has sent the slaveinput but the plugin is never activated on the slave hence no coverage. I can rsync pytest-cov over to the slave in same way as py / xdist already get done but unknown if I can activate the plugin. A separate but related issue: It would be good to have some way (cmd line / conftest / api) to tell xdist not to rsync py / xdist to the slave. The case is needed for the client code where it take care of all rsync so second rsync by xdist produce the warning about to two py and two xdist on path. :) From holger at merlinux.eu Mon Apr 26 11:29:39 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 26 Apr 2010 11:29:39 +0200 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <u2ycf54a0ac1004241906t57d5f488x87d8a421c4b446db@mail.gmail.com> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> <v2gcf54a0ac1004240701u4dfc94a1pcc3a5f160bb8828@mail.gmail.com> <u2ycf54a0ac1004241906t57d5f488x87d8a421c4b446db@mail.gmail.com> Message-ID: <20100426092939.GF22675@trillke.net> Hi Meme, On Sun, Apr 25, 2010 at 12:06 +1000, meme dough wrote: > Hi, > > > http://pypi.python.org/pypi/pytest-cover/ > > > > http://bitbucket.org/memedough/pytest-cover > > Renamed and moved to: > > http://pypi.python.org/pypi/pytest-cov > > http://bitbucket.org/memedough/pytest-cov > > It now coexists nicely with other plugins. I looked into the new plugin and have some immediate issues/questions: - ETOOMANYCMDLINEOPTIONS ... could most of the command line options go away if we rather rely on configurability through the config file? - can you add basic (functional or unit) tests? - is there a need to have a pytest_cov package instead of just a single module? cheers, holger From holger at merlinux.eu Mon Apr 26 11:41:09 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 26 Apr 2010 11:41:09 +0200 Subject: [py-dev] distributed test slave activation of external plugin In-Reply-To: <v2rcf54a0ac1004252149x7af83806te1d3ce921ff7599e@mail.gmail.com> References: <v2rcf54a0ac1004252149x7af83806te1d3ce921ff7599e@mail.gmail.com> Message-ID: <20100426094109.GG22675@trillke.net> Hi Meme, On Mon, Apr 26, 2010 at 14:49 +1000, meme dough wrote: > Hi, > > Is there any way to activate external plugins for distributed testing > on the slave side? it is usually done through "proper" installation or command line specification. > Currently pytest-cov requires that it is installed on the slave > version of python. This differs from pytest-xdist that will rsync > itself across, such that the only requirement on the slave side is > python. execnet and pytest-xdist are pure-python libs and known to work from just being transfered on a directory-level. This is often not true for packages requiring C-extensions. > If pytest-cov is not installed on the slave then the master has sent > the slaveinput but the plugin is never activated on the slave hence no > coverage. Yes, but currently you need to have the 'coverage' package installed anyway. > I can rsync pytest-cov over to the slave in same way as py / xdist > already get done but unknown if I can activate the plugin. We need a way to setup "remote" environments in a more local more controlled manner. This is a major step i'd like to take with xdist: configure automatic 'virtualenv' environments remotely, installing dependencies as needed. It would then be just a matter of ensuring 'pytest-cov' is in the dep list. Installed plugins are automatically discovered through the "standard" setuptools entry points. > A separate but related issue: It would be good to have some way (cmd > line / conftest / api) to tell xdist not to rsync py / xdist to the > slave. The case is needed for the client code where it take care of > all rsync so second rsync by xdist produce the warning about to two py > and two xdist on path. I suggest we move to use an 'xdist.ini' file where this could be specified. There we would also specify the virtualenvironments and global and specific package dependencies. The code that reads this config file and sets up environments should probably be useable outside of pytest as well - it'd be a nice aid for automatic config/install/control of app installs. holger From memedough at gmail.com Mon Apr 26 16:00:17 2010 From: memedough at gmail.com (meme dough) Date: Tue, 27 Apr 2010 00:00:17 +1000 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <20100426092939.GF22675@trillke.net> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> <v2gcf54a0ac1004240701u4dfc94a1pcc3a5f160bb8828@mail.gmail.com> <u2ycf54a0ac1004241906t57d5f488x87d8a421c4b446db@mail.gmail.com> <20100426092939.GF22675@trillke.net> Message-ID: <h2tcf54a0ac1004260700m2e7c342arbff84b15a5c6ec60@mail.gmail.com> Hi, > I looked into the new plugin and have some immediate issues/questions: > > - ETOOMANYCMDLINEOPTIONS ... > ?could most of the command line options go away > ?if we rather rely on configurability through the config file? Yes/No, while more cmd options with -h that maybe take up more space, it does give lots of control of coverage. I think this is good. It could move some of them to config file only but I don't think it good to remove completely. The good thing with pytest is the cmd options are auto done for conftest and env var which you can see with the --help-config with no work to do and consistent. Also all options have --cov prefix so clear and no clash. If only downside is more output on -h then seems small con. > - can you add basic (functional or unit) tests? Was early release with loose ends because people expressed an interest, so can test drive and have look. I have tests but are simple compared to real project testing with. Will commit when python versions issues are tidied up. > - is there a need to have a pytest_cov package instead of just a single module? Like this in preparation for rsync and activate plugin since better rsync dir I think, but your other email I like better. For now just require coverage / pytest-cov installed on slaves. But future have slave env setup through normal python way (distutils2 and virtualenv?). I will likely change to just module, but it seems more style thing so not sure if matters too much either way it done. :) From holger at merlinux.eu Mon Apr 26 18:03:31 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 26 Apr 2010 18:03:31 +0200 Subject: [py-dev] Testing and Coverage against different python interpreter versions In-Reply-To: <h2tcf54a0ac1004260700m2e7c342arbff84b15a5c6ec60@mail.gmail.com> References: <m2te1a84d571004201943x1fd749eam5e7c4c316d7140bc@mail.gmail.com> <k2hcf54a0ac1004210115k6ef8cb05zd7bd557a36fe7ee4@mail.gmail.com> <u2ve1a84d571004210123gb5a9349fhe760e981730d5312@mail.gmail.com> <v2gcf54a0ac1004240701u4dfc94a1pcc3a5f160bb8828@mail.gmail.com> <u2ycf54a0ac1004241906t57d5f488x87d8a421c4b446db@mail.gmail.com> <20100426092939.GF22675@trillke.net> <h2tcf54a0ac1004260700m2e7c342arbff84b15a5c6ec60@mail.gmail.com> Message-ID: <20100426160331.GH22675@trillke.net> On Tue, Apr 27, 2010 at 00:00 +1000, meme dough wrote: > > I looked into the new plugin and have some immediate issues/questions: > > > > - ETOOMANYCMDLINEOPTIONS ... > > ?could most of the command line options go away > > ?if we rather rely on configurability through the config file? > > Yes/No, while more cmd options with -h that maybe take up more space, > it does give lots of control of coverage. I think this is good. It > could move some of them to config file only but I don't think it good > to remove completely. > > The good thing with pytest is the cmd options are auto done for > conftest and env var which you can see with the --help-config with no > work to do and consistent. Also all options have --cov prefix so > clear and no clash. > > If only downside is more output on -h then seems small con. having around 10 options for one plugin makes it hard to know what the canonical usage is. And what happens if you have a mixed config-file/cmdline option situation? Also i doubt that e.g. being able to control output directories for html, xml, annotate needs to be that fine-grained, a '--cov-outputdir=...' is enough IMO. What about a generic --cov-option="...." that can be specified multiple times and can override config-ini values? This way you can (if you want to) still specify details on the command line but generally have a single place and way to specify config for coverage (in the ini file). > > - can you add basic (functional or unit) tests? > > Was early release with loose ends because people expressed an > interest, so can test drive and have look. > > I have tests but are simple compared to real project testing with. > Will commit when python versions issues are tidied up. > > > - is there a need to have a pytest_cov package instead of just a single module? > > Like this in preparation for rsync and activate plugin since better > rsync dir I think, but your other email I like better. For now just > require coverage / pytest-cov installed on slaves. But future have > slave env setup through normal python way (distutils2 and > virtualenv?). > > I will likely change to just module, but it seems more style thing so > not sure if matters too much either way it done. great, thanks for getting back on the issues. holger