From holger at merlinux.eu Sun Apr 12 23:49:23 2009 From: holger at merlinux.eu (holger krekel) Date: Sun, 12 Apr 2009 23:49:23 +0200 Subject: [py-dev] py-trunk on bitbucket / mercurial Message-ID: <20090412214923.GE8296@trillke.net> Hi everybody, since yesterday i am experimenting with using mercurial for py lib development. I have mirrored svn/py/trunk via "hgsvn" here: http://bitbucket.org/hpk42/py-trunk/ and work from there currently. You can follow the project there, fork off your own branches and also submit issues. cheers, holger -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From holger at merlinux.eu Sat Apr 18 23:10:50 2009 From: holger at merlinux.eu (holger krekel) Date: Sat, 18 Apr 2009 23:10:50 +0200 Subject: [py-dev] blog post on py.test beta and new docs Message-ID: <20090418211050.GD13377@trillke.net> Hi all, just published a new post with tons of links into some new documentation. Plugins and extensions still need better documentation and refinements but i hope you like it already. The blog post also talks about the imminent move to mercurial and other niceties. http://tinyurl.com/cdgfj2 cheers, holger -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From 5kycsae02 at sneakemail.com Mon Apr 20 06:07:58 2009 From: 5kycsae02 at sneakemail.com (Simon) Date: 20 Apr 2009 04:07:58 -0000 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet Message-ID: <7133-07379@sneakemail.com> Hi all, In my test harness, I often need to run the test code against a server. To do this I kick off a process on another machine (from the test scripts) to open up network connections. This remote process is run using py.execnet. What I find is that if the tests run to completion, everything is okay. However if I Control-C the test harness halfway through, the remote machine is left with an orphaned process: python -u -c exec input() It looks like the sshd instance has been killed but the python process is still alive. What is the correct way to shutdown an execnet connection so that there a no orphans left behind? Also what is a good way to hook this into the py.test infrastructure.? Simon From 5kycsae02 at sneakemail.com Tue Apr 21 03:25:59 2009 From: 5kycsae02 at sneakemail.com (Simon) Date: 21 Apr 2009 01:25:59 -0000 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet Message-ID: <21940-88024@sneakemail.com> This actually looks like an execnet issue, or maybe I'm not quite understanding how this should work. If I run the following code, I get a bunch of stuff left over. Aside from the ssh and shell instances, the main problem appears to be the python exec instance on the remote machine (in this example 'localhost'). If this process is killed, everything else seems to sort itself out. Calling the exit() method of the gateway doesn't seem to help. Nor does calling the close() method on the channel instance. ======================================= cmd = """ while True: pass """ import py g=py.execnet.SshGateway("localhost") c=g.remote_exec(cmd) ======================================= Simon From wam at cisco.com Wed Apr 22 16:54:31 2009 From: wam at cisco.com (William McVey) Date: Wed, 22 Apr 2009 10:54:31 -0400 Subject: [py-dev] Logging over execnet? Message-ID: <1240412071.10004.239.camel@talyn.cisco.com> I'm using py.execnet with the SshGateway and I'd like to fire off logging messages that get sent over the ssh connection. I was wondering if someone has already coded up a logging handler for this. Otherwise, I'm looking for suggestions on how to get logging messages sent from the remote side back to the local side. My goal would be that the logging events get sent over the ssh connection and get injected to the logging handler of the local/client side of the ssh connection. Also, is there is a "zero footprint" way of distributing the py library (specifically, the py.path component) to the remote side of a py.execnet? -- William P.S. I'd prefer the logging solution to use the standard library logging package, but if py.log makes exchanging messages over an execnet particularly easy, that would be fine by me as well. From 5kycsae02 at sneakemail.com Thu Apr 23 07:05:59 2009 From: 5kycsae02 at sneakemail.com (Simon) Date: Thu, 23 Apr 2009 15:05:59 +1000 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet In-Reply-To: <21940-88024@sneakemail.com> References: <21940-88024@sneakemail.com> Message-ID: <27026-01518@sneakemail.com> The core problem here appears to be that when the receive thread exits (because the ssh session has terminated), the main thread which is running the remote_exec is effectively orphaned. If the users remote exec task doesn't explicitly handle this, the main thread will stay open indefinitely. The attached patch corrects this by running the receive thread as the main thread and the remote_exec as a child daemon thread. Simon On 21/04/2009, at 11:25 , Simon 5kycsae02-at-sneakemail.com |py-dev| wrote: > This actually looks like an execnet issue, or maybe I'm not quite > understanding how this should work. > > If I run the following code, I get a bunch of stuff left over. Aside > from the ssh and shell instances, the main problem appears to be the > python exec instance on the remote machine (in this example > 'localhost'). If this process is killed, everything else seems to > sort itself out. > > Calling the exit() method of the gateway doesn't seem to help. Nor > does calling the close() method on the channel instance. > > ======================================= > cmd = """ > while True: > pass > """ > > import py > g=py.execnet.SshGateway("localhost") > c=g.remote_exec(cmd) > ======================================= > > Simon -------------- next part -------------- A non-text attachment was scrubbed... Name: gateway.py.diff Type: application/octet-stream Size: 1561 bytes Desc: not available URL: -------------- next part -------------- From phil at freehackers.org Thu Apr 23 18:00:34 2009 From: phil at freehackers.org (Philippe Fremy) Date: Thu, 23 Apr 2009 18:00:34 +0200 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? Message-ID: <49F090A2.3010101@freehackers.org> Hi, I am getting "(inconsistently failed then succeeded)" which are quite annoying. It's for code that checks conditions maniuplated by threads: t = threading.Thread( target=checker.start, args=(elcConfig, result) ) t.start() time.sleep( 0.1 ) checker.stop( False ) if not t.isAlive(): assert False if checker.isFinished(): assert False checker.join() > if t.isAlive(): assert False E AssertionError: (inconsistently failed then succeeded) I am playing a bit with race conditions here, but should that be a reason for my test to fail ? The conditions that I am checking arrive in the right order, only py.test is complaining. Any way I can get rid of "(inconsistently failed then succeeded)" ? cheers, Philippe From timothy.grant at gmail.com Thu Apr 23 23:07:37 2009 From: timothy.grant at gmail.com (Timothy Grant) Date: Thu, 23 Apr 2009 14:07:37 -0700 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: <49F090A2.3010101@freehackers.org> References: <49F090A2.3010101@freehackers.org> Message-ID: On Thu, Apr 23, 2009 at 9:00 AM, Philippe Fremy wrote: > > Hi, > > I am getting "(inconsistently failed then succeeded)" which are quite > annoying. It's for code that checks conditions maniuplated by threads: > > ? ? ? ?t = threading.Thread( target=checker.start, args=(elcConfig, > result) ?) > ? ? ? ?t.start() > > ? ? ? ?time.sleep( 0.1 ) > > ? ? ? ?checker.stop( False ) > > ? ? ? ?if not t.isAlive(): assert False > > ? ? ? ?if checker.isFinished(): assert False > > > > ? ? ? ?checker.join() > >> ? ? ? if t.isAlive(): assert False > > E ? ? ? AssertionError: (inconsistently failed then succeeded) > > > I am playing a bit with race conditions here, but should that be a > reason for my test to fail ? The conditions that I am checking arrive in > the right order, only py.test is complaining. > > Any way I can get rid of "(inconsistently failed then succeeded)" ? > > cheers, > > Philippe Philippe, I have found that storing the results of the call I want to test the result of, and then asserting the result generally eliminates these errors. For example: assert Foo() == 23 often caused me problems with that message, however: foo = Foo() assert foo == 23 would not. I am not sure why that is the case but I was able to resolve the issue using the second form. Hope that helps. -- Stand Fast, tjg. [Timothy Grant] From wam at cisco.com Thu Apr 23 23:10:24 2009 From: wam at cisco.com (William McVey) Date: Thu, 23 Apr 2009 17:10:24 -0400 Subject: [py-dev] Logging over execnet? In-Reply-To: <1240412071.10004.239.camel@talyn.cisco.com> References: <1240412071.10004.239.camel@talyn.cisco.com> Message-ID: <1240521024.10004.2317.camel@talyn.cisco.com> On Wed, 2009-04-22 at 10:54 -0400, William McVey wrote: > I'm using py.execnet with the SshGateway and I'd like to fire off > logging messages that get sent over the ssh connection. I was wondering > if someone has already coded up a logging handler for this. After a bit of frustration, I have remote logging over an execnet channel working, with no changes to the py.execnet code itself to support this. I had to resort to a nasty trick in order to get the channel to send and receive the logging.LogRecord objects. I think this could be made a lot cleaner if py.execnet were tweaked to allow defining a dictionary of namespaces to use when deserializing objects coming over a channel. If there is any interest in this by the py lib maintainers, I'll go ahead and fork the code using bitbucket, add this capability in (probably as a new method on the channel object) and issue a pull notification. Anyway, attached is a simple demo script that shows the ExecNetHandler logging handler in action. I'd love to hear feedback on this. -- William -------------- next part -------------- A non-text attachment was scrubbed... Name: test_log_execnet.py Type: text/x-python Size: 1913 bytes Desc: not available URL: From py-dev at tolomea.com Fri Apr 24 01:29:44 2009 From: py-dev at tolomea.com (Gordon Wrigley) Date: Fri, 24 Apr 2009 09:29:44 +1000 Subject: [py-dev] Logging over execnet? In-Reply-To: <1240521024.10004.2317.camel@talyn.cisco.com> References: <1240412071.10004.239.camel@talyn.cisco.com> <1240521024.10004.2317.camel@talyn.cisco.com> Message-ID: > If there is any interest in this by the py lib maintainers As a user I'm definitely interested in this, currently I deal with this style of problem by building my own marshaling layer over the channels, your "nasty trick" will allow me to simplify things quite a bit and your proposed change would improve them even further. Gordon On Fri, Apr 24, 2009 at 7:10 AM, William McVey wrote: > On Wed, 2009-04-22 at 10:54 -0400, William McVey wrote: >> I'm using py.execnet with the SshGateway and I'd like to fire off >> logging messages that get sent over the ssh connection. I was wondering >> if someone has already coded up a logging handler for this. > > After a bit of frustration, I have remote logging over an execnet > channel working, with no changes to the py.execnet code itself to > support this. I had to resort to a nasty trick in order to get the > channel to send and receive the logging.LogRecord objects. I think this > could be made a lot cleaner if py.execnet were tweaked to allow defining > a dictionary of namespaces to use when deserializing objects coming over > a channel. If there is any interest in this by the py lib maintainers, > I'll go ahead and fork the code using bitbucket, add this capability in > (probably as a new method on the channel object) and issue a pull > notification. > > Anyway, attached is a simple demo script that shows the ExecNetHandler > logging handler in action. I'd love to hear feedback on this. From nshepperd at gmail.com Fri Apr 24 12:38:48 2009 From: nshepperd at gmail.com (Neil Shepperd) Date: Fri, 24 Apr 2009 20:38:48 +1000 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: References: <49F090A2.3010101@freehackers.org> Message-ID: <1240569528.14532.16.camel@neil> Hi, > I have found that storing the results of the call I want to test the > result of, and then asserting the result generally eliminates these > errors. > > For example: > > assert Foo() == 23 > > often caused me problems with that message, however: > > foo = Foo() > assert foo == 23 > > would not. > > I am not sure why that is the case but I was able to resolve the issue > using the second form. If I understand it correctly, the test handling magic evaluates each part of the expression again, to display it in the fail traceback. For example, in the traceback of some test it will say: def test_xxx(): E assert Foo() == 23 > assert 0 == 23 If Foo() returns 0. It is run again get the value 0 to substitute into the error message. However if the second time it is run Foo() actually works correctly and returns 23, this value is obviously not correct, because if it were the assert would have passed. I suspect maybe py.test is evaluating t.isAlive() again to substitute into the if-expression, resulting in confusion when the result changes to True. If that is the case, you might have to do something like checker.stop( False ) is_alive = t.isAlive() is_finished = checker.isFinished() if not is_alive: assert False if is_finished: assert False Hope this made sense, Neil From phil at freehackers.org Fri Apr 24 14:45:36 2009 From: phil at freehackers.org (Philippe Fremy) Date: Fri, 24 Apr 2009 14:45:36 +0200 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: <1240569528.14532.16.camel@neil> References: <49F090A2.3010101@freehackers.org> <1240569528.14532.16.camel@neil> Message-ID: <49F1B470.7020006@freehackers.org> Neil Shepperd wrote: > Hi, > >> I have found that storing the results of the call I want to test the >> result of, and then asserting the result generally eliminates these >> errors. >> >> For example: >> >> assert Foo() == 23 >> >> often caused me problems with that message, however: >> >> foo = Foo() >> assert foo == 23 >> >> would not. >> >> I am not sure why that is the case but I was able to resolve the issue >> using the second form. > If I understand it correctly, the test handling magic evaluates each > part of the expression again, to display it in the fail traceback. > For example, in the traceback of some test it will say: > > def test_xxx(): > E assert Foo() == 23 >> assert 0 == 23 > > If Foo() returns 0. It is run again get the value 0 to substitute into > the error message. However if the second time it is run Foo() actually > works correctly and returns 23, this value is obviously not correct, > because if it were the assert would have passed. > > I suspect maybe py.test is evaluating t.isAlive() again to substitute > into the if-expression, resulting in confusion when the result changes > to True. If that is the case, you might have to do something like > > checker.stop( False ) > is_alive = t.isAlive() > is_finished = checker.isFinished() > if not is_alive: assert False > if is_finished: assert False > > > Hope this made sense, Thanks, it makes sense now on the cause of the error. This ought to be documented somewhere in py.test cheers, Philippe From holger at merlinux.eu Mon Apr 27 18:32:18 2009 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 Apr 2009 18:32:18 +0200 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet In-Reply-To: <27026-01518@sneakemail.com> References: <21940-88024@sneakemail.com> <27026-01518@sneakemail.com> Message-ID: <20090427163218.GN13377@trillke.net> Hi Simon, On Thu, Apr 23, 2009 at 15:05 +1000, Simon wrote: > The core problem here appears to be that when the receive thread exits > (because the ssh session has terminated), the main thread which is > running the remote_exec is effectively orphaned. If the users remote > exec task doesn't explicitly handle this, the main thread will stay open > indefinitely. > > The attached patch corrects this by running the receive thread as the > main thread and the remote_exec as a child daemon thread. thanks for your investigations and patch! When I run your example script with your patch and immediately do a "ps aux" afterwards i still see the subprocess but it dies after around 10 seconds. Do you also see this behaviour? I'd like py.execnet remain able to remotely invoke GUI applications or other apps which need the main thread for their event loop. So I am wondering if there is a different way to achieve the same result of a clean shutdown. I just created a branch here: http://bitbucket.org/hpk42/py-shutdown/ and intend to first to write some tests. Please feel free to follow and contribute. cheers, holger > Simon > > On 21/04/2009, at 11:25 , Simon 5kycsae02-at-sneakemail.com |py-dev| > wrote: > >> This actually looks like an execnet issue, or maybe I'm not quite >> understanding how this should work. >> >> If I run the following code, I get a bunch of stuff left over. Aside >> from the ssh and shell instances, the main problem appears to be the >> python exec instance on the remote machine (in this example >> 'localhost'). If this process is killed, everything else seems to sort >> itself out. >> >> Calling the exit() method of the gateway doesn't seem to help. Nor >> does calling the close() method on the channel instance. >> >> ======================================= >> cmd = """ >> while True: >> pass >> """ >> >> import py >> g=py.execnet.SshGateway("localhost") >> c=g.remote_exec(cmd) >> ======================================= >> >> Simon > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From holger at merlinux.eu Mon Apr 27 19:25:16 2009 From: holger at merlinux.eu (holger krekel) Date: Mon, 27 Apr 2009 19:25:16 +0200 Subject: [py-dev] Logging over execnet? In-Reply-To: <1240521024.10004.2317.camel@talyn.cisco.com> References: <1240412071.10004.239.camel@talyn.cisco.com> <1240521024.10004.2317.camel@talyn.cisco.com> Message-ID: <20090427172516.GO13377@trillke.net> Hi William, On Thu, Apr 23, 2009 at 17:10 -0400, William McVey wrote: > On Wed, 2009-04-22 at 10:54 -0400, William McVey wrote: > > I'm using py.execnet with the SshGateway and I'd like to fire off > > logging messages that get sent over the ssh connection. I was wondering > > if someone has already coded up a logging handler for this. I think it is a fine usecase. > After a bit of frustration, I have remote logging over an execnet > channel working, with no changes to the py.execnet code itself to > support this. I had to resort to a nasty trick in order to get the > channel to send and receive the logging.LogRecord objects. I think this > could be made a lot cleaner if py.execnet were tweaked to allow defining > a dictionary of namespaces to use when deserializing objects coming over > a channel. If there is any interest in this by the py lib maintainers, > I'll go ahead and fork the code using bitbucket, add this capability in > (probably as a new method on the channel object) and issue a pull > notification. > Anyway, attached is a simple demo script that shows the ExecNetHandler > logging handler in action. I'd love to hear feedback on this. you could put the "remote_code" into a module and pass the imported module to remote_exec - this allows to keep syntax coloring and also allows to more easily unittest it. Maybe that would alleviate some of the frustration? Also you could maybe just send a dictionary of instance values and call a Logrecord object with it as **kwargs. As you say yourself, your code works on the presumtion that py.execnet uses repr/eval for sending and receiving of objects. That may change soon and would break your code. A more general solution for sending non-marshallable objects is to have some pickling layer on top of current channels. For use with distributed testing, i've implemented a "PickleChannel", see here: http://bitbucket.org/hpk42/py-trunk/src/c90f1bf8bac4/py/test/dist/mypickle.py with unit and functional tests here: http://bitbucket.org/hpk42/py-trunk/src/c90f1bf8bac4/py/test/dist/testing/test_mypickle.py This code also uses an additional concept "ImmutablePickle" which restricts object pickling in the following way: If Process A sends an object OBJ to process B and process B sends it back (either directly or as an attribute of another object) then A will get a reference to OBJ and not a fresh copy. Usually pickle would create a copy in process A. I've found this preserving of object identity useful for distributing testing tasks. Obviously one could also use plain pickle, maybe by introducing a pickling option to remote_exec() that could have different values like e.g. "ipickle", "pickle", "repr" etc. If you are up to coding this in a fork i'd be happy to review it. As a first step, having "pickle" and a default that reflects the current situation would be fine and should help your use case. I'd probably keep the layering as it probably keeps code changes to a minimum. cheers, holger > -- William > import py, time, logging > > remote_code = """ > import logging > > class FakeLogRecord(object): > "Fakes a LogRecord's repr for passing over an execnet channel" > def __init__(self, log): > self.log = log > > def __repr__(self): > l = self.log > # Serialization of objects going over a channel happens by calling > # repr() on the object. Deserialization happens with an > # eval(mesg, {}). With a minimal namespace, I need to handle the > # importing of the module myself as an expression encoded in the repr > # of the object. Currently, I ignore specified traceback objects > # attached to a LogRecord via exc_info=True. > return "__import__('logging').LogRecord(%r, %r, %r, %r, %r, %r, ())" % ( > l.name, l.levelno, l.pathname, l.lineno, l.msg, l.args) > > > class ExecNetHandler(logging.Handler): > "Send logging messages over an py.execnet connection" > > def __init__(self, channel): > logging.Handler.__init__(self) > self.channel = channel > > def emit(self, record): > obj = FakeLogRecord(record) > self.channel.send(obj) > > # A testrun > log = logging.getLogger("remote") > log.setLevel(logging.DEBUG) > log.addHandler(ExecNetHandler(channel)) > > log.debug("Starting up") > channel.send("Look ma, I'm home") > log.info("I'm running on the remote side") > try: > raise RuntimeError("just an exception") > except: > log.error("Somebody set up us the bomb", exc_info=True) > log.debug("Closing down") > """ > > logging.basicConfig(format="%(name)s: %(levelname)s: %(message)s") > remote_log = logging.getLogger("remote") > handler = logging.StreamHandler() > handler.setLevel(logging.INFO) > remote_log.addHandler(handler) > > > gw = py.execnet.PopenGateway() > channel = gw.remote_exec(remote_code) > > time.sleep(2) > for mesg in channel: > if isinstance(mesg, logging.LogRecord): > remote_log.handle(mesg) > continue > print "From the remote side:", `mesg` > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From 5kycsae02 at sneakemail.com Tue Apr 28 03:44:10 2009 From: 5kycsae02 at sneakemail.com (Simon) Date: Tue, 28 Apr 2009 11:44:10 +1000 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet In-Reply-To: <20090427163218.GN13377@trillke.net> References: <21940-88024@sneakemail.com> <27026-01518@sneakemail.com> <20090427163218.GN13377@trillke.net> Message-ID: <14700-47063@sneakemail.com> Hi Holger, On 28/04/2009, at 02:32 , holger krekel holger-at-merlinux.eu |py-dev| wrote: > Hi Simon, > > On Thu, Apr 23, 2009 at 15:05 +1000, Simon wrote: >> The core problem here appears to be that when the receive thread >> exits >> (because the ssh session has terminated), the main thread which is >> running the remote_exec is effectively orphaned. If the users remote >> exec task doesn't explicitly handle this, the main thread will stay >> open >> indefinitely. >> >> The attached patch corrects this by running the receive thread as the >> main thread and the remote_exec as a child daemon thread. > > thanks for your investigations and patch! > > When I run your example script with your patch > and immediately do a "ps aux" afterwards > i still see the subprocess but it dies > after around 10 seconds. Do you also see this > behaviour? Yes that is correct. When the main thread which handles communications starts to shutdown, it will signal the child thread running the exec by pushing a None into the queue. Then the main thread will attempt to join the child thread with a 10 second timeout. If the child thread reacts and closed cleanly, then it will shutdown and the main thread can complete immediately. If the child thread fails to shutdown cleanly within 10 seconds, the main thread will exit and the daemonised child thread will be cleaned up by the OS. The 10 second delay is somewhat arbitrary, we put it in to allow the thread some time to shutdown cleanly (in the event that it hasn't actually hung and is just a little busy). > I'd like py.execnet remain able to remotely invoke > GUI applications or other apps which need the main > thread for their event loop. So I am wondering if there is a > different way to achieve the same result of a clean shutdown. Do you mean that you want the applications to outlive the execnet connection? If that is so, it should be trivial to make the daemonisation optional which would mean that the exec process could continue on beyond comms shutdown. Simon > I just created a branch here: > > http://bitbucket.org/hpk42/py-shutdown/ > > and intend to first to write some tests. > Please feel free to follow and contribute. > > cheers, > holger > > >> Simon >> >> On 21/04/2009, at 11:25 , Simon 5kycsae02-at-sneakemail.com |py-dev| >> wrote: >> >>> This actually looks like an execnet issue, or maybe I'm not quite >>> understanding how this should work. >>> >>> If I run the following code, I get a bunch of stuff left over. Aside >>> from the ssh and shell instances, the main problem appears to be the >>> python exec instance on the remote machine (in this example >>> 'localhost'). If this process is killed, everything else seems to >>> sort >>> itself out. >>> >>> Calling the exit() method of the gateway doesn't seem to help. Nor >>> does calling the close() method on the channel instance. >>> >>> ======================================= >>> cmd = """ >>> while True: >>> pass >>> """ >>> >>> import py >>> g=py.execnet.SshGateway("localhost") >>> c=g.remote_exec(cmd) >>> ======================================= >>> >>> Simon >> > > >> > >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev > > > -- > Metaprogramming, Python, Testing: http://tetamap.wordpress.com > Python, PyPy, pytest contracting: http://merlinux.eu From holger at merlinux.eu Tue Apr 28 11:05:57 2009 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 Apr 2009 11:05:57 +0200 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet In-Reply-To: <14700-47063@sneakemail.com> References: <21940-88024@sneakemail.com> <27026-01518@sneakemail.com> <20090427163218.GN13377@trillke.net> <14700-47063@sneakemail.com> Message-ID: <20090428090557.GA11963@trillke.net> Hi Simon, On Tue, Apr 28, 2009 at 11:44 +1000, Simon wrote: > On 28/04/2009, at 02:32 , holger krekel holger-at-merlinux.eu |py-dev| > wrote: > > > Hi Simon, > > > > On Thu, Apr 23, 2009 at 15:05 +1000, Simon wrote: > >> The core problem here appears to be that when the receive thread > >> exits > >> (because the ssh session has terminated), the main thread which is > >> running the remote_exec is effectively orphaned. If the users remote > >> exec task doesn't explicitly handle this, the main thread will stay > >> open > >> indefinitely. > >> > >> The attached patch corrects this by running the receive thread as the > >> main thread and the remote_exec as a child daemon thread. > > > > thanks for your investigations and patch! > > > > When I run your example script with your patch > > and immediately do a "ps aux" afterwards > > i still see the subprocess but it dies > > after around 10 seconds. Do you also see this > > behaviour? > Yes that is correct. When the main thread which handles communications > starts to shutdown, it will signal the child thread running the exec > by pushing a None into the queue. Then the main thread will attempt to > join the child thread with a 10 second timeout. If the child thread > reacts and closed cleanly, then it will shutdown and the main thread > can complete immediately. If the child thread fails to shutdown > cleanly within 10 seconds, the main thread will exit and the > daemonised child thread will be cleaned up by the OS. > > The 10 second delay is somewhat arbitrary, we put it in to allow the > thread some time to shutdown cleanly (in the event that it hasn't > actually hung and is just a little busy). > > > > I'd like py.execnet remain able to remotely invoke > > GUI applications or other apps which need the main > > thread for their event loop. So I am wondering if there is a > > different way to achieve the same result of a clean shutdown. > Do you mean that you want the applications to outlive the execnet > connection? If that is so, it should be trivial to make the > daemonisation optional which would mean that the exec process could > continue on beyond comms shutdown. i meant that GUI threads need the main thread to run their event loop. With your patch execution will always run in sub threads on the remote side. The issue you mention is another one. Are you aware of the gateway.remote_init_threads() method, btw? It will run all subsequent execution in sub daemon threads on the remote side. This should effectively do the same as your patch. I'd be curious if that solves your problem. Independently, we probably need to parametrize gateway.exit(). If exit() is triggered implicitely by process exit or garbage collection it should immediately kill the remote process including all its execution threads, i.e. with a zero timeout. If one wants to be more careful with remote execution threads one would need to call exit() with a timeout paramter or "no timeout" which would keep the process alive until all execution finishes. Does this make sense to you? Needs to work for both the main-thread and multi-exec-thread configuration, of course. cheers, holger > Simon > > > I just created a branch here: > > > > http://bitbucket.org/hpk42/py-shutdown/ > > > > and intend to first to write some tests. > > Please feel free to follow and contribute. > > > > cheers, > > holger > > > > > >> Simon > >> > >> On 21/04/2009, at 11:25 , Simon 5kycsae02-at-sneakemail.com |py-dev| > >> wrote: > >> > >>> This actually looks like an execnet issue, or maybe I'm not quite > >>> understanding how this should work. > >>> > >>> If I run the following code, I get a bunch of stuff left over. Aside > >>> from the ssh and shell instances, the main problem appears to be the > >>> python exec instance on the remote machine (in this example > >>> 'localhost'). If this process is killed, everything else seems to > >>> sort > >>> itself out. > >>> > >>> Calling the exit() method of the gateway doesn't seem to help. Nor > >>> does calling the close() method on the channel instance. > >>> > >>> ======================================= > >>> cmd = """ > >>> while True: > >>> pass > >>> """ > >>> > >>> import py > >>> g=py.execnet.SshGateway("localhost") > >>> c=g.remote_exec(cmd) > >>> ======================================= > >>> > >>> Simon > >> > > > > > >> > > > >> _______________________________________________ > >> py-dev mailing list > >> py-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/py-dev > > > > > > -- > > Metaprogramming, Python, Testing: http://tetamap.wordpress.com > > Python, PyPy, pytest contracting: http://merlinux.eu > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From holger at merlinux.eu Tue Apr 28 12:14:54 2009 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 Apr 2009 12:14:54 +0200 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: <49F1B470.7020006@freehackers.org> References: <49F090A2.3010101@freehackers.org> <1240569528.14532.16.camel@neil> <49F1B470.7020006@freehackers.org> Message-ID: <20090428101454.GB11963@trillke.net> Hi Philippe, Neil, On Fri, Apr 24, 2009 at 14:45 +0200, Philippe Fremy wrote: > Neil Shepperd wrote: > > Hi, > > > >> I have found that storing the results of the call I want to test the > >> result of, and then asserting the result generally eliminates these > >> errors. > >> > >> For example: > >> > >> assert Foo() == 23 > >> > >> often caused me problems with that message, however: > >> > >> foo = Foo() > >> assert foo == 23 > >> > >> would not. > >> > >> I am not sure why that is the case but I was able to resolve the issue > >> using the second form. > > If I understand it correctly, the test handling magic evaluates each > > part of the expression again, to display it in the fail traceback. > > For example, in the traceback of some test it will say: > > > > def test_xxx(): > > E assert Foo() == 23 > >> assert 0 == 23 > > > > If Foo() returns 0. It is run again get the value 0 to substitute into > > the error message. However if the second time it is run Foo() actually > > works correctly and returns 23, this value is obviously not correct, > > because if it were the assert would have passed. > > > > I suspect maybe py.test is evaluating t.isAlive() again to substitute > > into the if-expression, resulting in confusion when the result changes > > to True. If that is the case, you might have to do something like > > > > checker.stop( False ) > > is_alive = t.isAlive() > > is_finished = checker.isFinished() > > if not is_alive: assert False > > if is_finished: assert False > > > > > > Hope this made sense, > > Thanks, it makes sense now on the cause of the error. This ought to be > documented somewhere in py.test i agree, the error message is slightly mysterious. what about this: def test_inconsistent(): def f(l=[1,0]): return l.pop() > assert f() E AssertionError: (assert failed but re-evaluating the assert expression for printing intermediate values lead to a True value. advise: avoid side-effects in assert expressions or use --nomagic) It's a bit longish but i doubt people encountering the problem would know where to lookup things in the documentation. Any other suggestion for the error message? cheers, holger > cheers, > > Philippe > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From lac at openend.se Tue Apr 28 12:54:18 2009 From: lac at openend.se (Laura Creighton) Date: Tue, 28 Apr 2009 12:54:18 +0200 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: Message from holger krekel of "Tue, 28 Apr 2009 12:14:54 +0200." <20090428101454.GB11963@trillke.net> References: <49F090A2.3010101@freehackers.org> <1240569528.14532.16.camel@neil> <49F1B470.7020006@freehackers.org> <20090428101454.GB11963@trillke.net> Message-ID: <200904281054.n3SAsIIs013660@theraft.openend.se> In a message of Tue, 28 Apr 2009 12:14:54 +0200, holger krekel writes: >i agree, the error message is slightly mysterious. what about this: > > def test_inconsistent(): > def f(l=[1,0]): > return l.pop() >> assert f() >E AssertionError: (assert failed but re-evaluating the >assert expression for printing intermediate values lead to a >True value. advise: avoid side-effects in assert expressions >or use --nomagic) > >It's a bit longish but i doubt people encountering the problem >would know where to lookup things in the documentation. >Any other suggestion for the error message? > >cheers, > >holger I think that if you are going to be that long, then you might as well be a bit longer. I also think that the word you were looking for was 'advice', which rhymes with mice, and is a noun, and not 'advise' which rhymes with 'surprise' and is a verb, which means to give somebody advice. I like 'suggest' better, but maybe that is just me. When I've run into this before it is in the context of doing TDD with children. They don't know what a side effect is, and I don't think that your error message explains that there is a real problem with the way that they wrote the assert in the first place. At any rate, the puzzling thing around here was that it passed in the second case, not that it failed the first time around, but it left students with the idea that maybe their code was correct, when it wasn't. I also think that it is more natural to talk about asserts failing and then passing, or being false and then true, rather than failing and then becoming true. So here's my attempt at an error message: E: AssertionError: Evaluating the assert has undesirable side-effects. The assert fails when the test is first run, but when subsequently re-evaluated for printing intermediate values, it passes. Suggest: rewriting the assert expression in a way that avoids side-effects, or use --nomagic) Laura From phil at freehackers.org Tue Apr 28 13:33:58 2009 From: phil at freehackers.org (Philippe Fremy) Date: Tue, 28 Apr 2009 13:33:58 +0200 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: <20090428101454.GB11963@trillke.net> References: <49F090A2.3010101@freehackers.org> <1240569528.14532.16.camel@neil> <49F1B470.7020006@freehackers.org> <20090428101454.GB11963@trillke.net> Message-ID: <49F6E9A6.4010508@freehackers.org> holger krekel wrote: > Hi Philippe, Neil, > > On Fri, Apr 24, 2009 at 14:45 +0200, Philippe Fremy wrote: >> Neil Shepperd wrote: >>> Hi, >>> >>>> I have found that storing the results of the call I want to test the >>>> result of, and then asserting the result generally eliminates these >>>> errors. >>>> >>>> For example: >>>> >>>> assert Foo() == 23 >>>> >>>> often caused me problems with that message, however: >>>> >>>> foo = Foo() >>>> assert foo == 23 >>>> >>>> would not. >>>> >>>> I am not sure why that is the case but I was able to resolve the issue >>>> using the second form. >>> If I understand it correctly, the test handling magic evaluates each >>> part of the expression again, to display it in the fail traceback. >>> For example, in the traceback of some test it will say: >>> >>> def test_xxx(): >>> E assert Foo() == 23 >>>> assert 0 == 23 >>> If Foo() returns 0. It is run again get the value 0 to substitute into >>> the error message. However if the second time it is run Foo() actually >>> works correctly and returns 23, this value is obviously not correct, >>> because if it were the assert would have passed. >>> >>> I suspect maybe py.test is evaluating t.isAlive() again to substitute >>> into the if-expression, resulting in confusion when the result changes >>> to True. If that is the case, you might have to do something like >>> >>> checker.stop( False ) >>> is_alive = t.isAlive() >>> is_finished = checker.isFinished() >>> if not is_alive: assert False >>> if is_finished: assert False >>> >>> >>> Hope this made sense, >> Thanks, it makes sense now on the cause of the error. This ought to be >> documented somewhere in py.test > > i agree, the error message is slightly mysterious. what about this: > > def test_inconsistent(): > def f(l=[1,0]): > return l.pop() >> assert f() > E AssertionError: (assert failed but re-evaluating the > assert expression for printing intermediate values lead to a > True value. advise: avoid side-effects in assert expressions > or use --nomagic) I like it :-) As long as it's documented somewhere what's the cause and how to get around it, I'm fine. cheers, Philippe From phil at freehackers.org Tue Apr 28 13:37:37 2009 From: phil at freehackers.org (Philippe Fremy) Date: Tue, 28 Apr 2009 13:37:37 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level Message-ID: <49F6EA81.2070106@freehackers.org> Hi, I was wondering if there is a possibility to have or emulate setup/teardown method for an entire directory of tests ? I have an expensive setup/teardown method, which I share through about 10 modules, each with about 20 tests. I would save time if I could do the setup/teardown just for that directory. cheers, Philippe From holger at merlinux.eu Tue Apr 28 13:40:36 2009 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 Apr 2009 13:40:36 +0200 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: <200904281054.n3SAsIIs013660@theraft.openend.se> References: <49F090A2.3010101@freehackers.org> <1240569528.14532.16.camel@neil> <49F1B470.7020006@freehackers.org> <20090428101454.GB11963@trillke.net> <200904281054.n3SAsIIs013660@theraft.openend.se> Message-ID: <20090428114036.GI11963@trillke.net> On Tue, Apr 28, 2009 at 12:54 +0200, Laura Creighton wrote: > In a message of Tue, 28 Apr 2009 12:14:54 +0200, holger krekel writes: > >i agree, the error message is slightly mysterious. what about this: > > > > def test_inconsistent(): > > def f(l=[1,0]): > > return l.pop() > >> assert f() > >E AssertionError: (assert failed but re-evaluating the > >assert expression for printing intermediate values lead to a > >True value. advise: avoid side-effects in assert expressions > >or use --nomagic) > > > >It's a bit longish but i doubt people encountering the problem > >would know where to lookup things in the documentation. > >Any other suggestion for the error message? > > > >cheers, > > > >holger > > I think that if you are going to be that long, then you might as well > be a bit longer. I also think that the word you were looking for was > 'advice', which rhymes with mice, and is a noun, and not 'advise' > which rhymes with 'surprise' and is a verb, which means to give > somebody advice. I like 'suggest' better, but maybe that is just me. ok :) > When I've run into this before it is in the context of doing TDD with > children. They don't know what a side effect is, and I don't think > that your error message explains that there is a real problem with the > way that they wrote the assert in the first place. At any rate, the > puzzling thing around here was that it passed in the second case, > not that it failed the first time around, but it left students with > the idea that maybe their code was correct, when it wasn't. ok, that means to me to try to avoid the term "side-effect". > I also think that it is more natural to talk about asserts failing and > then passing, or being false and then true, rather than failing and > then becoming true. > > So here's my attempt at an error message: > > E: AssertionError: Evaluating the assert has undesirable side-effects. > The assert fails when the test is first run, but when subsequently > re-evaluated for printing intermediate values, it passes. Suggest: > rewriting the assert expression in a way that avoids side-effects, or > use --nomagic) taking your valid points into account, i'd i'd like to go with something like this (re-posting the func example): def test_inconsistent(): def f(l=[1,0]): return l.pop() > assert f() E: AssertionError: (assertion failed, but when it was re-run for printing intermediate values, it did not fail. Suggestions: compute assert expression before the assert or use --nomagic) fine to you? thanks & cheers, holger From holger at merlinux.eu Tue Apr 28 13:55:56 2009 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 Apr 2009 13:55:56 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level In-Reply-To: <49F6EA81.2070106@freehackers.org> References: <49F6EA81.2070106@freehackers.org> Message-ID: <20090428115556.GJ11963@trillke.net> Hi Philippe, On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote: > Hi, > > I was wondering if there is a possibility to have or emulate > setup/teardown method for an entire directory of tests ? > > I have an expensive setup/teardown method, which I share through about > 10 modules, each with about 20 tests. I would save time if I could do > the setup/teardown just for that directory. do you have a setup_module()/teardown_module() pair that you import into all your tests? Do you essentially want to setup the test resource only once per test-run? cheers, holger > cheers, > > Philippe > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From lac at openend.se Tue Apr 28 15:03:19 2009 From: lac at openend.se (Laura Creighton) Date: Tue, 28 Apr 2009 15:03:19 +0200 Subject: [py-dev] Gitting rid of "(inconsistently failed then succeeded)" ? In-Reply-To: Message from holger krekel of "Tue, 28 Apr 2009 13:40:36 +0200." <20090428114036.GI11963@trillke.net> References: <49F090A2.3010101@freehackers.org> <1240569528.14532.16.camel@neil> <49F1B470.7020006@freehackers.org> <20090428101454.GB11963@trillke.net> <200904281054.n3SAsIIs013660@theraft.openend.se> <20090428114036.GI11963@trillke.net> Message-ID: <200904281303.n3SD3JxT025552@theraft.openend.se> In a message of Tue, 28 Apr 2009 13:40:36 +0200, holger krekel writes: >taking your valid points into account, i'd >i'd like to go with something like this (re-posting the func example): > > def test_inconsistent(): > def f(l=[1,0]): return l.pop() >> assert f() >E: AssertionError: (assertion failed, but when it was re-run for >printing intermediate values, it did not fail. Suggestions: >compute assert expression before the assert or use --nomagic) > >fine to you? >thanks & cheers, >holger Looks great to me. :) Laura From phil at freehackers.org Tue Apr 28 17:09:02 2009 From: phil at freehackers.org (Philippe Fremy) Date: Tue, 28 Apr 2009 17:09:02 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level In-Reply-To: <20090428115556.GJ11963@trillke.net> References: <49F6EA81.2070106@freehackers.org> <20090428115556.GJ11963@trillke.net> Message-ID: <49F71C0E.3000800@freehackers.org> holger krekel wrote: > Hi Philippe, Hi Holger, > > On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote: >> Hi, >> >> I was wondering if there is a possibility to have or emulate >> setup/teardown method for an entire directory of tests ? >> >> I have an expensive setup/teardown method, which I share through about >> 10 modules, each with about 20 tests. I would save time if I could do >> the setup/teardown just for that directory. > > do you have a setup_module()/teardown_module() pair that > you import into all your tests? Exactly. > Do you essentially want to setup the test resource > only once per test-run? Once per session would be actually sufficient for this specific case today, but the need is once per test directory. To be a bit more explicit, in my current software development, I have both unit tests for functionality in App1 that I develop myself, and functional tests for a wrapper around App2, which is an external application : base + app1 + tests + many unit tests + app2_wrapper + external_app2 + tests + many functional tests for the app2 wrapper, requiring to setup a resource for app2 to run. I launch py.test from the base directory. I have an expensive setup/teardown for the functional tests of the app2 wrapper, which I would like to share at the app2_wrapper/tests directory level. Ideally, if I run only tests from app1, I don't run that expensive setup/teardown. cheers, Philippe From holger at merlinux.eu Tue Apr 28 20:02:00 2009 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 Apr 2009 20:02:00 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level In-Reply-To: <49F71C0E.3000800@freehackers.org> References: <49F6EA81.2070106@freehackers.org> <20090428115556.GJ11963@trillke.net> <49F71C0E.3000800@freehackers.org> Message-ID: <20090428180200.GO11963@trillke.net> Hi Philippe, your described use case makes lots of sense to me. I coded an example which i hope fits it. It uses the new "local plugins" (i.e. plugins defined in a conftest.py) and funcargs, if you don't know about them yet i hope this is good to skim/read first: http://codespeak.net/py/trunk/test/funcargs.html Here is the example: http://bitbucket.org/hpk42/py-trunk/src/tip/example/funcarg/lazysetup/ using py-trunk (probably also works with the 1.0.0b1, haven't checked) in the lazysetup directory you can now do py.test sub1 # will wait 5 seconds because test # functions access the setup defined in # conftest.py py.test sub2 # will immediately run as the "setup" # funcarg is not requested The idea for this conftest.py implementation is simple: setup the funcarg when first needed and only tear it down when the test process exits. does this make sense to you? feel free to play around and ask questions - I'd then put the above example into the "tutorial" example section of the funcarg doc. One advantage of the above approach is that you do not need to do anything in your test modules anymore (no boilerplate importing of setup_module etc.) than requesting the object you want to setup. cheers, holger On Tue, Apr 28, 2009 at 17:09 +0200, Philippe Fremy wrote: > holger krekel wrote: > > Hi Philippe, > > Hi Holger, > > > > > On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote: > >> Hi, > >> > >> I was wondering if there is a possibility to have or emulate > >> setup/teardown method for an entire directory of tests ? > >> > >> I have an expensive setup/teardown method, which I share through about > >> 10 modules, each with about 20 tests. I would save time if I could do > >> the setup/teardown just for that directory. > > > > do you have a setup_module()/teardown_module() pair that > > you import into all your tests? > > Exactly. > > > Do you essentially want to setup the test resource > > only once per test-run? > > Once per session would be actually sufficient for this specific case > today, but the need is once per test directory. > > To be a bit more explicit, in my current software development, I have > both unit tests for functionality in App1 that I develop myself, and > functional tests for a wrapper around App2, which is an external > application : > > base > + app1 > + tests > + many unit tests > + app2_wrapper > + external_app2 > + tests > + many functional tests for the app2 wrapper, requiring to setup > > a resource for app2 to run. > > I launch py.test from the base directory. > > I have an expensive setup/teardown for the functional tests of the app2 > wrapper, which I would like to share at the app2_wrapper/tests directory > level. > > Ideally, if I run only tests from app1, I don't run that expensive > setup/teardown. > > cheers, > > Philippe > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From 5kycsae02 at sneakemail.com Wed Apr 29 03:08:01 2009 From: 5kycsae02 at sneakemail.com (Simon) Date: Wed, 29 Apr 2009 11:08:01 +1000 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet In-Reply-To: <20090428090557.GA11963@trillke.net> References: <21940-88024@sneakemail.com> <27026-01518@sneakemail.com> <20090427163218.GN13377@trillke.net> <14700-47063@sneakemail.com> <20090428090557.GA11963@trillke.net> Message-ID: <7026-00862@sneakemail.com> Hi Holger, On 28/04/2009, at 19:05 , holger krekel holger-at-merlinux.eu |py-dev| wrote: > i meant that GUI threads need the main thread to run their > event loop. With your patch execution will always run in > sub threads on the remote side. Just a quick question, why does the GUIs event loop need to run in the gateways main thread? I understand that all the widget initialisation and event handling has to be done from the same thread, but I don't understand why this can't be a created thread. Simon From holger at merlinux.eu Wed Apr 29 09:31:04 2009 From: holger at merlinux.eu (holger krekel) Date: Wed, 29 Apr 2009 09:31:04 +0200 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet In-Reply-To: <7026-00862@sneakemail.com> References: <21940-88024@sneakemail.com> <27026-01518@sneakemail.com> <20090427163218.GN13377@trillke.net> <14700-47063@sneakemail.com> <20090428090557.GA11963@trillke.net> <7026-00862@sneakemail.com> Message-ID: <20090429073103.GQ11963@trillke.net> Hi Simon, On Wed, Apr 29, 2009 at 11:08 +1000, Simon wrote: > Hi Holger, > > On 28/04/2009, at 19:05 , holger krekel holger-at-merlinux.eu |py-dev| > wrote: > > > i meant that GUI threads need the main thread to run their > > event loop. With your patch execution will always run in > > sub threads on the remote side. > > Just a quick question, why does the GUIs event loop need to run in the > gateways main thread? I understand that all the widget initialisation > and event handling has to be done from the same thread, but I don't > understand why this can't be a created thread. i don't know. It's something that people and web pages told/tell me. I am hardly doing GUI programming myself. Is it maybe because only the main thread receives signals? did you try out remote_init_threads(1)? If that doesn't work for you i'd like to fix it or apply your patch in some way. cheers, holger > Simon > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From phil at freehackers.org Thu Apr 30 10:22:01 2009 From: phil at freehackers.org (Philippe Fremy) Date: Thu, 30 Apr 2009 10:22:01 +0200 Subject: [py-dev] Keyboard interrupt causes unclean shutdown in py.test & py.execnet In-Reply-To: <20090429073103.GQ11963@trillke.net> References: <21940-88024@sneakemail.com> <27026-01518@sneakemail.com> <20090427163218.GN13377@trillke.net> <14700-47063@sneakemail.com> <20090428090557.GA11963@trillke.net> <7026-00862@sneakemail.com> <20090429073103.GQ11963@trillke.net> Message-ID: <49F95FA9.6040607@freehackers.org> holger krekel wrote: > Hi Simon, > > On Wed, Apr 29, 2009 at 11:08 +1000, Simon wrote: >> Hi Holger, >> >> On 28/04/2009, at 19:05 , holger krekel holger-at-merlinux.eu |py-dev| >> wrote: >> >>> i meant that GUI threads need the main thread to run their >>> event loop. With your patch execution will always run in >>> sub threads on the remote side. >> Just a quick question, why does the GUIs event loop need to run in the >> gateways main thread? I understand that all the widget initialisation >> and event handling has to be done from the same thread, but I don't >> understand why this can't be a created thread. > > i don't know. It's something that people and web pages > told/tell me. I am hardly doing GUI programming myself. > Is it maybe because only the main thread receives signals? I can't give the true explanation either, but it seems that it creates awfully complicated code as soon as you stop assuming that the GUI does not run in the main thread. And that's for a core component of a GUI toolkit where things are quite complicated already. It seems to be the explanation behind this decision, for example, for Qt: "In Qt, one thread is always the GUI or event thread. This is the thread that creates a QApplication object and calls QApplication::exec(). This is also the initial thread that calls main() at program start. This thread is the only thread that is allowed to perform GUI operations, including generating and receiving events from the window system." cheers, Philippe From phil at freehackers.org Thu Apr 30 10:22:54 2009 From: phil at freehackers.org (Philippe Fremy) Date: Thu, 30 Apr 2009 10:22:54 +0200 Subject: [py-dev] py.test : setup / teardown at the directory level In-Reply-To: <20090428180200.GO11963@trillke.net> References: <49F6EA81.2070106@freehackers.org> <20090428115556.GJ11963@trillke.net> <49F71C0E.3000800@freehackers.org> <20090428180200.GO11963@trillke.net> Message-ID: <49F95FDE.1010003@freehackers.org> Hi Holger, Thanks for looking into this. I am quite busy until mid next week so it will take me a few days before I get back to you. cheers, Philippe holger krekel wrote: > Hi Philippe, > > your described use case makes lots of sense to me. > > I coded an example which i hope fits it. > > It uses the new "local plugins" (i.e. plugins defined in a > conftest.py) and funcargs, if you don't know about them > yet i hope this is good to skim/read first: > http://codespeak.net/py/trunk/test/funcargs.html > > Here is the example: > > http://bitbucket.org/hpk42/py-trunk/src/tip/example/funcarg/lazysetup/ > > using py-trunk (probably also works with the 1.0.0b1, haven't checked) > in the lazysetup directory you can now do > > py.test sub1 # will wait 5 seconds because test > # functions access the setup defined in > # conftest.py > > py.test sub2 # will immediately run as the "setup" > # funcarg is not requested > > The idea for this conftest.py implementation is simple: > setup the funcarg when first needed and only tear it down > when the test process exits. > > does this make sense to you? feel free to play around > and ask questions - I'd then put the above example into > the "tutorial" example section of the funcarg doc. > > One advantage of the above approach is that you do not > need to do anything in your test modules anymore > (no boilerplate importing of setup_module etc.) > than requesting the object you want to setup. > > cheers, > holger > > On Tue, Apr 28, 2009 at 17:09 +0200, Philippe Fremy wrote: >> holger krekel wrote: >>> Hi Philippe, >> Hi Holger, >> >>> On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote: >>>> Hi, >>>> >>>> I was wondering if there is a possibility to have or emulate >>>> setup/teardown method for an entire directory of tests ? >>>> >>>> I have an expensive setup/teardown method, which I share through about >>>> 10 modules, each with about 20 tests. I would save time if I could do >>>> the setup/teardown just for that directory. >>> do you have a setup_module()/teardown_module() pair that >>> you import into all your tests? >> Exactly. >> >>> Do you essentially want to setup the test resource >>> only once per test-run? >> Once per session would be actually sufficient for this specific case >> today, but the need is once per test directory. >> >> To be a bit more explicit, in my current software development, I have >> both unit tests for functionality in App1 that I develop myself, and >> functional tests for a wrapper around App2, which is an external >> application : >> >> base >> + app1 >> + tests >> + many unit tests >> + app2_wrapper >> + external_app2 >> + tests >> + many functional tests for the app2 wrapper, requiring to setup >> >> a resource for app2 to run. >> >> I launch py.test from the base directory. >> >> I have an expensive setup/teardown for the functional tests of the app2 >> wrapper, which I would like to share at the app2_wrapper/tests directory >> level. >> >> Ideally, if I run only tests from app1, I don't run that expensive >> setup/teardown. >> >> cheers, >> >> Philippe >> >> _______________________________________________ >> py-dev mailing list >> py-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/py-dev >> >