From martin.blais at gmail.com Fri Apr 1 04:23:08 2005 From: martin.blais at gmail.com (Martin Blais) Date: Thu, 31 Mar 2005 21:23:08 -0500 Subject: [py-dev] py.test and --session, re-run failures only Message-ID: <8393fff05033118237484edc0@mail.gmail.com> hi guys my first post on this list. thanks for py.test, it's very nice, i'm in the process of converting my cr*ppy unittest hacks into something a little cleaner by using py.test. in the past, i wrote some testing infrastructure for a large C++ project in the past, i hope i can contribute something to this project in the future. 1. so i'm checking out py.test and i run across this: --session run a test session/rerun only failing tests. "rerun only failing tests" lets me think that somehow the session is recording somewhere (e.g. in a file) which tests have failed and only reruns those tests. in fact, fiddling in the code and running it, i realized that it just runs a subprocess and monitors file changes (by polling), and then automatically reruns the tests. perhaps the help string should be clarified. in any case, it would be a cool feature to be able to record which tests fail and then to be able to rerun just the ones which failed in the past. we used to have that in my previous job, and it was very useful, we had LOTS of tests, and running them was very slow, so just rerunniing the failures was a great timesaver. 2. i would like to be able to say "run just this one test", by specifying the method, e.g. test_search.py:TestSearch.test_simple, or perhaps some kind simpler pattern matching. this is why i started fiddling in the code. i would probably modify the driver to filter down the wanted methods once the collector has passed. any comments on how to go about this welcome. thanks, cheers, From hpk at trillke.net Sat Apr 2 02:27:07 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 2 Apr 2005 02:27:07 +0200 Subject: [py-dev] py.test and --session, re-run failures only In-Reply-To: <8393fff05033118237484edc0@mail.gmail.com> References: <8393fff05033118237484edc0@mail.gmail.com> Message-ID: <20050402002707.GJ23818@solar.trillke.net> Hi Martin, nice to have you here! I hope you had a good flight back to Canada. On Thu, Mar 31, 2005 at 21:23 -0500, Martin Blais wrote: > hi guys > > my first post on this list. thanks for py.test, it's very nice, i'm > in the process of converting my cr*ppy unittest hacks into something a > little cleaner by using py.test. in the past, i wrote some testing > infrastructure for a large C++ project in the past, i hope i can > contribute something to this project in the future. sounds good! > 1. > so i'm checking out py.test and i run across this: > > --session run a test session/rerun only failing tests. > > "rerun only failing tests" lets me think that somehow the session is > recording somewhere (e.g. in a file) which tests have failed and only > reruns those tests. in fact, fiddling in the code and running it, i > realized that it just runs a subprocess and monitors file changes (by > polling), and then automatically reruns the tests. perhaps the help > string should be clarified. Yes, although it's not exactly easy to put this in a short help text. If you have a suggestion please send it. > in any case, it would be a cool feature to be able to record which > tests fail and then to be able to rerun just the ones which failed in > the past. we used to have that in my previous job, and it was very > useful, we had LOTS of tests, and running them was very slow, so just > rerunniing the failures was a great timesaver. Yes, recording failing tests is a good idea. The problem is determining a) the location to store failing test information b) the format in which "test addresses" should be stored a) points to a more general problem (having per-user storage for the py lib and py.test in particular) that i'd like to solve in the near future. b) is more or less handlded in the "collect" branch which i will need to merge real soon now. Basically a "test address" consists of a rootpath and a list of names. It's important that such "addresses" will work nice for the upcoming doctest support. > 2. > i would like to be able to say "run just this one test", by specifying > the method, e.g. test_search.py:TestSearch.test_simple, or perhaps > some kind simpler pattern matching. this is why i started fiddling in > the code. Very valid use case and i have a plan how to satisfy it ... > i would probably modify the driver to filter down the wanted methods > once the collector has passed. any comments on how to go about this > welcome. ... first of all, note that the collect branch contains the future version of py.test. Second, i have added a chapter in the py.test documentation that describes future features (still open to discussion). Have a look at: http://codespeak.net/py/current/doc/test.html#future-planned-features-of-py-test cheers, holger From martin.blais at gmail.com Sat Apr 2 02:44:25 2005 From: martin.blais at gmail.com (Martin Blais) Date: Fri, 1 Apr 2005 19:44:25 -0500 Subject: [py-dev] py.test and --session, re-run failures only In-Reply-To: <20050402002707.GJ23818@solar.trillke.net> References: <8393fff05033118237484edc0@mail.gmail.com> <20050402002707.GJ23818@solar.trillke.net> Message-ID: <8393fff0504011644bac7386@mail.gmail.com> On Apr 1, 2005 7:27 PM, holger krekel wrote: > Hi Martin, > > nice to have you here! I hope you had a good flight back to Canada. thx i drove back with mr docutils > On Thu, Mar 31, 2005 at 21:23 -0500, Martin Blais wrote: > > hi guys > > > > my first post on this list. thanks for py.test, it's very nice, i'm > > in the process of converting my cr*ppy unittest hacks into something a > > little cleaner by using py.test. in the past, i wrote some testing > > infrastructure for a large C++ project in the past, i hope i can > > contribute something to this project in the future. > > sounds good! > > > 1. > > so i'm checking out py.test and i run across this: > > > > --session run a test session/rerun only failing tests. > > > > "rerun only failing tests" lets me think that somehow the session is > > recording somewhere (e.g. in a file) which tests have failed and only > > reruns those tests. in fact, fiddling in the code and running it, i > > realized that it just runs a subprocess and monitors file changes (by > > polling), and then automatically reruns the tests. perhaps the help > > string should be clarified. > > Yes, although it's not exactly easy to put this in a short help text. > If you have a suggestion please send it. > > > in any case, it would be a cool feature to be able to record which > > tests fail and then to be able to rerun just the ones which failed in > > the past. we used to have that in my previous job, and it was very > > useful, we had LOTS of tests, and running them was very slow, so just > > rerunniing the failures was a great timesaver. > > Yes, recording failing tests is a good idea. The problem is determining > > a) the location to store failing test information > b) the format in which "test addresses" should be stored > > a) points to a more general problem (having per-user storage for the py > lib and py.test in particular) that i'd like to solve in the near future. how about a simple option for now? this way a user can have multiple session files if desired. personally, when a cmdline gets too long, i always stash it in a Makefile anyway so lots of options is not a problem. the makefile can determine whatever policy for selecting a session file location. > b) is more or less handlded in the "collect" branch which i will need to > merge real soon now. Basically a "test address" consists of > a rootpath and a list of names. It's important that such "addresses" > will work nice for the upcoming doctest support. i was thinking we could just use the funky extpy path syntax (like it is seen with --collectonly). > > 2. > > i would like to be able to say "run just this one test", by specifying > > the method, e.g. test_search.py:TestSearch.test_simple, or perhaps > > some kind simpler pattern matching. this is why i started fiddling in > > the code. > > Very valid use case and i have a plan how to satisfy it ... > > > i would probably modify the driver to filter down the wanted methods > > once the collector has passed. any comments on how to go about this > > welcome. > > ... first of all, note that the collect branch contains the future version > of py.test. Second, i have added a chapter in the py.test documentation > that describes future features (still open to discussion). Have a look at: > > http://codespeak.net/py/current/doc/test.html#future-planned-features-of-py-test oh, while we're taking a shot at the future, another feature that i used to have was the ability to compare the test output to a fixed file, we called this "expect files", because the test was expected to generate specific output. might not be useful now but you could note that in the futures document as well, for some kinds of tests this can be a quick-n-dirty way to write a test. dump something on the output, inspect it to make sure it's allright, dump that into an expect file. could be easy to implement since you already capture the output text. note that these kinds of tests fail on outputs sensitive to numerical (im)precision. and another one: for C++ code which can crash, a timeout option is absolutely *essential*. cheers, From ianb at colorstudy.com Sat Apr 2 02:48:51 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 01 Apr 2005 18:48:51 -0600 Subject: [py-dev] py.test and --session, re-run failures only In-Reply-To: <8393fff0504011644bac7386@mail.gmail.com> References: <8393fff05033118237484edc0@mail.gmail.com> <20050402002707.GJ23818@solar.trillke.net> <8393fff0504011644bac7386@mail.gmail.com> Message-ID: <424DEBF3.9080304@colorstudy.com> Martin Blais wrote: > oh, while we're taking a shot at the future, another feature that i > used to have was the ability to compare the test output to a fixed > file, we called this "expect files", because the test was expected to > generate specific output. might not be useful now but you could note > that in the futures document as well, for some kinds of tests this can > be a quick-n-dirty way to write a test. dump something on the output, > inspect it to make sure it's allright, dump that into an expect file. > could be easy to implement since you already capture the output text. > note that these kinds of tests fail on outputs sensitive to numerical > (im)precision. That's called doctest ;) -- of course better integration of that is also in the future at this point. > and another one: for C++ code which can crash, a timeout option is > absolutely *essential*. This could potentially be implemented as a decorator, like: @py.test.timeout(10) def test_cpp_code(): ... def timeout(seconds): def replacement_func(*args, **kw): some stuff with alarm signals and whatnot... return replacement_func Of course, you can't apply a decorator globally with an option, but it's probably only in specific areas where you'd need this anyway. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From martin.blais at gmail.com Sat Apr 2 02:52:50 2005 From: martin.blais at gmail.com (Martin Blais) Date: Fri, 1 Apr 2005 19:52:50 -0500 Subject: [py-dev] py.test and --session, re-run failures only In-Reply-To: <424DEBF3.9080304@colorstudy.com> References: <8393fff05033118237484edc0@mail.gmail.com> <20050402002707.GJ23818@solar.trillke.net> <8393fff0504011644bac7386@mail.gmail.com> <424DEBF3.9080304@colorstudy.com> Message-ID: <8393fff0504011652702a2a60@mail.gmail.com> On Apr 1, 2005 7:48 PM, Ian Bicking wrote: > Martin Blais wrote: > > oh, while we're taking a shot at the future, another feature that i > > used to have was the ability to compare the test output to a fixed > > file, we called this "expect files", because the test was expected to > > generate specific output. might not be useful now but you could note > > that in the futures document as well, for some kinds of tests this can > > be a quick-n-dirty way to write a test. dump something on the output, > > inspect it to make sure it's allright, dump that into an expect file. > > could be easy to implement since you already capture the output text. > > note that these kinds of tests fail on outputs sensitive to numerical > > (im)precision. > > That's called doctest ;) -- of course better integration of that is also > in the future at this point. well... not quite. it's you have 1000 lines of output i wouldn't want to have to skip that in the source code. > > and another one: for C++ code which can crash, a timeout option is > > absolutely *essential*. > > This could potentially be implemented as a decorator, like: > > @py.test.timeout(10) > def test_cpp_code(): > ... > > def timeout(seconds): > def replacement_func(*args, **kw): > some stuff with alarm signals and whatnot... > return replacement_func > > Of course, you can't apply a decorator globally with an option, but it's > probably only in specific areas where you'd need this anyway. IMHO that's generic enough an option to add globally. besides i can hear lots of people cringe at decorators in the background... From hpk at trillke.net Sat Apr 2 13:26:08 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 2 Apr 2005 13:26:08 +0200 Subject: [py-dev] anybody using conftest.py? Message-ID: <20050402112608.GK23818@solar.trillke.net> hi especially py.test users, i am wondering if any of you are using (the undocumented) "conftest.py" files within your projects where you use py.test. If so, please describe briefly what you are doing with it. I want to merge the collect-branch real soon now and this will involve changes to customizations people might have been doing via conftest.py files. If you don't know what i am talking about, fine: forget it :-) One case where i will need to care is PyPy which already uses test-customization heavily. cheers, holger From jan at balster.info Mon Apr 4 00:19:40 2005 From: jan at balster.info (Jan Balster) Date: Mon, 04 Apr 2005 00:19:40 +0200 Subject: [py-dev] py.test tkinter frontend Message-ID: <42506BFC.7080209@balster.info> Hello, i finished my rewrite of a tkinter-frontend for py.test. what's the best way to share it? i can post a screenshot and a tar.gz file (35kb). some informations about the package: it lives under py/test/gui and works only in the py-collect branch. it provides a tree view for all tests. every test can be executed just by double clicking it. the frontend watches all files and directories specified at the command line. if a test fails, a text widget shows a report similar to the current report by py.test (it uses TerminalDriver). all filenames in a report are links and open a text editor (see tkgui.py/TkMain.launch_editor) to show the point of failure. if a test fails the frontend calls tkinter.bell(), so you can hear if your changes break some code (should be an option). --collectonly, --nocapture, --showlocals , --fulltrace, --nomagic, should work as expected. todo: * documentation, documentation * more tests ;-) * replace hack for config.py and conftest.py !! * patch drive.py/Driver.runone -> call finish after the captured output is set (see py.test.gui.guidriver.py/GuiDriver.runone) * improve performance of tree widget when updating test results!! i would appreciate your feedback and comments! thank you! jan From hpk at trillke.net Sun Apr 3 23:30:54 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 3 Apr 2005 23:30:54 +0200 Subject: [py-dev] py.test tkinter frontend In-Reply-To: <42506BFC.7080209@balster.info> References: <42506BFC.7080209@balster.info> Message-ID: <20050403213054.GO23818@solar.trillke.net> Hello Jan, On Mon, Apr 04, 2005 at 00:19 +0200, Jan Balster wrote: > Hello, > > i finished my rewrite of a tkinter-frontend for py.test. what's the best > way to share it? hum, what about you send me your desired accound name + ssh public key privately and you just commit it? > ... > some informations about the package: cool! > todo: > > * documentation, documentation > > * more tests ;-) That is actually important and i know that writing tests for interactive applications ca be hard especially if done after the fact ... > * replace hack for config.py and conftest.py !! i'd like to see what you have done and how it compares to the hacks we have done for PyPy :-) > ... > i would appreciate your feedback and comments! i am still somewhat ill and have personal issues to care for at the moment ... so if others can give your code a try (once it is checked in) that would be great ... cheers & many thanks, holger From hpk at trillke.net Mon Apr 4 16:44:44 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 4 Apr 2005 16:44:44 +0200 Subject: [py-dev] Re: [py-svn] r10293 - in py/branch/py-collect: bin test/tkinter test/tkinter/icons In-Reply-To: <20050404153241.C117F27BC2@code1.codespeak.net> References: <20050404153241.C117F27BC2@code1.codespeak.net> Message-ID: <20050404144444.GW23818@solar.trillke.net> Hi Jan, On Mon, Apr 04, 2005 at 17:32 +0200, jan at codespeak.net wrote: > Author: jan > Date: Mon Apr 4 17:32:41 2005 > New Revision: 10293 > > Added: > py/branch/py-collect/bin/py.test.tkinter (contents, props changed) > py/branch/py-collect/test/tkinter/ > ... thanks for committing, i have a few preliminary comments ... - would you mind respecting http://codespeak.net/py/current/doc/coding-style.html and in particular "no plural names in directory and file names"? - would it be hard to get your code 2.2 (at least 2.3) compatible? the py lib and py.test aim to run on python2.2 and upwards ... cheers, holger From roy at panix.com Tue Apr 5 14:50:37 2005 From: roy at panix.com (Roy Smith) Date: Tue, 5 Apr 2005 08:50:37 -0400 Subject: [py-dev] Minor doc problem with py.test Message-ID: <44f73b3091555a1c2920d950ceb9f4d8@panix.com> I'm new to the list, so I'm not sure what the right procedure is for reporting these things. I assume somebody will set me straight if this list isn't it :-) In http://codespeak.net/py/current/doc/test.html#automatic-collection-of- tests-on-all-levels, it says, "The automated test collection process collects files from all subdirectories with a leading test_ in their filename". I had to read it a few times to understand that "with a leading test_" applied to the individual filenames, not the directory names. It might be clearer to say something like: "The automated test collection process walks the current directory and all subdirectories (except those starting with a ".") and collects files with a leading test_ in their filename" From hpk at trillke.net Tue Apr 5 15:10:23 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 5 Apr 2005 15:10:23 +0200 Subject: [py-dev] Minor doc problem with py.test In-Reply-To: <44f73b3091555a1c2920d950ceb9f4d8@panix.com> References: <44f73b3091555a1c2920d950ceb9f4d8@panix.com> Message-ID: <20050405131023.GC23818@solar.trillke.net> Hi Roy, On Tue, Apr 05, 2005 at 08:50 -0400, Roy Smith wrote: > I'm new to the list, so I'm not sure what the right procedure is for > reporting these things. I assume somebody will set me straight if this > list isn't it :-) That's perfectly fine. > In http://codespeak.net/py/current/doc/test.html#automatic-collection-of- > tests-on-all-levels, it says, "The automated test collection process > collects files from all subdirectories with a leading test_ in their > filename". I had to read it a few times to understand that "with a > leading test_" applied to the individual filenames, not the directory > names. It might be clearer to say something like: > > "The automated test collection process walks the current directory and > all subdirectories (except those starting with a ".") and collects > files with a leading test_ in their filename" I have clarified the appropriate sections taking your suggestions into account. I hope that helps. cheers, holger From roy at panix.com Tue Apr 5 15:15:09 2005 From: roy at panix.com (Roy Smith) Date: Tue, 5 Apr 2005 09:15:09 -0400 Subject: [py-dev] Does each test method get called with a new instance? Message-ID: Is the control flow example at http://codespeak.net/py/current/doc/test.html#example-for-managing- state-at-module-class-and-method-level correct? It implies that instance.test_42() and instance.test_23() both get called with the same instance. I would have thought a new instance would be created for each call. Hmmmm, a little experimenting shows that it is indeed correct. This seems counter-intuitive. I've always considered it a basic maxim of unit tests that each test ran in a pristine environment. What's the logic here? From hpk at trillke.net Tue Apr 5 15:19:33 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 5 Apr 2005 15:19:33 +0200 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: References: Message-ID: <20050405131933.GE23818@solar.trillke.net> Hi Roy, On Tue, Apr 05, 2005 at 09:15 -0400, Roy Smith wrote: > Is the control flow example at > http://codespeak.net/py/current/doc/test.html#example-for-managing- > state-at-module-class-and-method-level correct? It implies that > instance.test_42() and instance.test_23() both get called with the same > instance. I would have thought a new instance would be created for > each call. > > Hmmmm, a little experimenting shows that it is indeed correct. This > seems counter-intuitive. I've always considered it a basic maxim of > unit tests that each test ran in a pristine environment. What's the > logic here? If you want to customize a test class instance per test method invocation you should use def setup_method(self, meth): ... def teardown_method(self, meth): ... to do that. Relying on __init__() and __del__() for such state management is not sensible because Python - the language - does not guarantee timely finalization/cleanup via __del__(). It is thus more sensible to advertise setup_method/teardown_method as the way to reliably handle test state management. HTH, holger From roy at panix.com Tue Apr 5 23:11:56 2005 From: roy at panix.com (Roy Smith) Date: Tue, 05 Apr 2005 17:11:56 -0400 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050405131933.GE23818@solar.trillke.net> References: <20050405131933.GE23818@solar.trillke.net> Message-ID: <4252FF1C.10207@panix.com> holger krekel wrote: >Hi Roy, > >On Tue, Apr 05, 2005 at 09:15 -0400, Roy Smith wrote: > > >>Is the control flow example at >>http://codespeak.net/py/current/doc/test.html#example-for-managing- >>state-at-module-class-and-method-level correct? It implies that >>instance.test_42() and instance.test_23() both get called with the same >>instance. I would have thought a new instance would be created for >>each call. >> >>Hmmmm, a little experimenting shows that it is indeed correct. This >>seems counter-intuitive. I've always considered it a basic maxim of >>unit tests that each test ran in a pristine environment. What's the >>logic here? >> >> > >If you want to customize a test class instance per test method invocation >you should use > > def setup_method(self, meth): ... > def teardown_method(self, meth): ... > >to do that. Relying on __init__() and __del__() for such state >management is not sensible because Python - the language - does >not guarantee timely finalization/cleanup via __del__(). It is >thus more sensible to advertise setup_method/teardown_method >as the way to reliably handle test state management. > >HTH, > > holger > > OK, but this seems a little strange to me. Most of the time, all I'm really interested in is that I've got a fresh instance of the class being tested. While it may be true that old instances (from previous test methods) may not have been finalized yet, that's only going to be of concern when there's something significant to finalize (i.e. close a file handle or something like that). In those specific (but rare) cases, I could then use the teardown() hook to clean things up, and most of the time I could get away without a teardown(). This way, it seems like I need to do: def setup_method (self, method): self.testInstance = ClassUnderTest() def teardown_method (self, method): del self.testInstance to get a clean instance for every test method. This I get for free with unittest. And of course with this, everyplace I'd normally use self.foo in my test methods, I'd now have to use self.testInstance.foo. It seems like things would be a lot simplier if a new instance was created for each test method call. You could still use the teardown_method for force finalizations when needed, but most of the time you wouldn't have to bother. Or am I just not getting it? From hpk at trillke.net Tue Apr 5 23:34:11 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 5 Apr 2005 23:34:11 +0200 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <4252FF1C.10207@panix.com> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> Message-ID: <20050405213411.GM23818@solar.trillke.net> Hi Roy, On Tue, Apr 05, 2005 at 17:11 -0400, Roy Smith wrote: > Most of the time, all I'm really interested in is that I've got a fresh > instance of the class being tested. While it may be true that old > instances (from previous test methods) may not have been finalized yet, > that's only going to be of concern when there's something significant to > finalize (i.e. close a file handle or something like that). In those > specific (but rare) cases, I could then use the teardown() hook to clean > things up, and most of the time I could get away without a teardown(). I think i understand your resoning. > This way, it seems like I need to do: > > def setup_method (self, method): > self.testInstance = ClassUnderTest() > > def teardown_method (self, method): > del self.testInstance > > to get a clean instance for every test method. > > This I get for free with unittest. And of course with this, > everyplace I'd normally use self.foo in my test methods, I'd > now have to use self.testInstance.foo. Not quite, instead of writing def __init__(self): ... you just write def setup_method(self, method): ... and be done with it. This makes it very explicit in which scope your code will be invoked, doesn't it? It seems a common use case that people want to customize per test method invocation (after all, that's what unittest's setUp() does) so there is a need for setup_method/teardown_method, anyway. But why offer a second way (per __init__ which doesn't know about the method) to do the same thing in a more restricted way? > Or am I just not getting it? test state management is actually a pretty involved issue. py.test tries to provide very explicit hooks where their names leave as few doubts about when and in which way they will be invoked. cheers, holger From ianb at colorstudy.com Tue Apr 5 23:39:02 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 05 Apr 2005 16:39:02 -0500 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050405213411.GM23818@solar.trillke.net> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> Message-ID: <42530576.4050801@colorstudy.com> holger krekel wrote: > It seems a common use case that people want to customize per test > method invocation (after all, that's what unittest's setUp() does) > so there is a need for setup_method/teardown_method, anyway. > But why offer a second way (per __init__ which doesn't know about > the method) to do the same thing in a more restricted way? Where this becomes interesting, is when you do: class Test: def __init__(self, input, output): self.input, self.output = input, output ... def test_Test(): for fn in os.listdir('data/'): if fn.startswith('input'): yield Test, fn, fn.replace('input', 'output') Can you do that now? I think Test would be automatically instantiated and throw an error...? Well, one way or another you *should* be able to do that. And then class initialization is usefully orthogonal to test setup. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From hpk at trillke.net Wed Apr 6 00:09:26 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 6 Apr 2005 00:09:26 +0200 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <42530576.4050801@colorstudy.com> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <42530576.4050801@colorstudy.com> Message-ID: <20050405220926.GP23818@solar.trillke.net> On Tue, Apr 05, 2005 at 16:39 -0500, Ian Bicking wrote: > holger krekel wrote: > >It seems a common use case that people want to customize per test > >method invocation (after all, that's what unittest's setUp() does) > >so there is a need for setup_method/teardown_method, anyway. > >But why offer a second way (per __init__ which doesn't know about > >the method) to do the same thing in a more restricted way? > > Where this becomes interesting, is when you do: > > class Test: > > def __init__(self, input, output): > self.input, self.output = input, output > ... > > def test_Test(): > for fn in os.listdir('data/'): > if fn.startswith('input'): > yield Test, fn, fn.replace('input', 'output') > > Can you do that now? I think Test would be automatically instantiated > and throw an error...? There is no "magic instantiation" for generative tests. (Sometimes i think the existence of "py.magic" makes people afraid that magic is happening at all sorts of places where in fact it wants to indicate that magic is rather localized :-) Effectively your above example would just lead to testing the successful invocation of the __init__ constructor. FWIW, a more full blown example of what you might want to do is: def test_Test(): datadir = py.magic.autopath.dirpath('data') # [1] for inputfn in datadir.listdir('input*'): newbasename = inputfn.basename.replace('input', 'output') outputfn = inputfn.new(basename=newbasename) yield Test(inputfn, outputfn) # this assumes you want to run __call__ for the actual test Indeed, you may not need a class at all but can just yield a test function for each input/output pair: yield check_input_output, intputfn, outputfn cheers, holger [1] there it is again, the dreaded but convenient py.magic namespace: py.magic.autopath() looks at your module's __file__ and gives you a path object so that path.dirpath('data') gets you to the 'data' directory accompanying your test file, irrespective of the current working directory. From ianb at colorstudy.com Wed Apr 6 00:24:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 05 Apr 2005 17:24:11 -0500 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050405220926.GP23818@solar.trillke.net> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <42530576.4050801@colorstudy.com> <20050405220926.GP23818@solar.trillke.net> Message-ID: <4253100B.6090609@colorstudy.com> holger krekel wrote: > Effectively your above example would just lead to testing the > successful invocation of the __init__ constructor. FWIW, a > more full blown example of what you might want to do is: > > def test_Test(): > datadir = py.magic.autopath.dirpath('data') # [1] > for inputfn in datadir.listdir('input*'): > newbasename = inputfn.basename.replace('input', 'output') > outputfn = inputfn.new(basename=newbasename) > yield Test(inputfn, outputfn) > # this assumes you want to run __call__ for the actual test Hrm... I see why my version wouldn't work like I want, but this isn't actually what I'd want either. I think there's a very useful pattern in this case, where you'd want Test(...) to be an instance that would be tested with py.test's full machinery -- meaning that all the test* methods could be run, setup_method, etc. This is kind of coming back to me wanting the process of turning objects (like functions, classes, etc) into test items to be recursive, applied after generator functions parameterize the tests. At least, I *think* that is what I wanted at one point... I am fickle and my desires are often short-lived ;) (Oh, BTW, both SQLObject and WSGIKit use py.test for all their testing now -- this particular issue hasn't quite come up, but I think it may as I increasingly build custom frameworks for their respective tests. But it's pretty low-priority compared to other things) > [1] there it is again, the dreaded but convenient py.magic namespace: > py.magic.autopath() looks at your module's __file__ and gives > you a path object so that path.dirpath('data') gets you > to the 'data' directory accompanying your test file, irrespective > of the current working directory. Yeah, I was just being sloppy in my example -- in real life I'd be using os.path.dirname(__file__) and whatnot, all of which makes Python look really really ugly ;) I'm still feeling confused about whether to stick with os.path -- stupid but familiar and doesn't put any runtime py lib requirements on code -- or Orendorf's path module -- a little crufty, but I really like the string subclass, because people can ignore its specialness if they feel more comfortable with os.path -- or py.path, which I'll admit is probably a more elegant interface. One way or the other, I feel like I have to *accept* string paths as arguments, and path objects seem less compelling when I'm always instantiating new ones and whatnot. Hrm. Not that I don't hate os.path, mind you. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From hpk at trillke.net Wed Apr 6 00:51:32 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 6 Apr 2005 00:51:32 +0200 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <4253100B.6090609@colorstudy.com> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <42530576.4050801@colorstudy.com> <20050405220926.GP23818@solar.trillke.net> <4253100B.6090609@colorstudy.com> Message-ID: <20050405225132.GQ23818@solar.trillke.net> On Tue, Apr 05, 2005 at 17:24 -0500, Ian Bicking wrote: > holger krekel wrote: > >Effectively your above example would just lead to testing the > >successful invocation of the __init__ constructor. FWIW, a > >more full blown example of what you might want to do is: > > > > def test_Test(): > > datadir = py.magic.autopath.dirpath('data') # [1] > > for inputfn in datadir.listdir('input*'): > > newbasename = inputfn.basename.replace('input', 'output') > > outputfn = inputfn.new(basename=newbasename) > > yield Test(inputfn, outputfn) > > # this assumes you want to run __call__ for the actual test > > Hrm... I see why my version wouldn't work like I want, but this isn't > actually what I'd want either. I think there's a very useful pattern in > this case, where you'd want Test(...) to be an instance that would be > tested with py.test's full machinery -- meaning that all the test* > methods could be run, setup_method, etc. Ah, i see where you are pushing me. There might be ways to do this, at least with the current py-collect branch. I freely admit that generative tests via yield started out as a hack to accomodate criticism from some crazy guy, named Ian. The existence of this hack (and people appreciating it) was one of the reasons for the py-collect branch which lead to the collection & running process to converge significantly. > This is kind of coming back to me wanting the process of > turning objects (like functions, classes, etc) into test > items to be recursive, applied after generator functions > parameterize the tests. At least, I *think* that is what I > wanted at one point... I am fickle and my desires are often > short-lived ;) For the time being, i would like to keep generative tests as simple as possible. > (Oh, BTW, both SQLObject and WSGIKit use py.test for all their testing > now -- this particular issue hasn't quite come up, but I think it may as > I increasingly build custom frameworks for their respective tests. But > it's pretty low-priority compared to other things) Good :-) > >[1] there it is again, the dreaded but convenient py.magic namespace: > > py.magic.autopath() looks at your module's __file__ and gives > > you a path object so that path.dirpath('data') gets you > > to the 'data' directory accompanying your test file, irrespective > > of the current working directory. > > Yeah, I was just being sloppy in my example -- in real life I'd be using > os.path.dirname(__file__) and whatnot, all of which makes Python look > really really ugly ;) I'm still feeling confused about whether to stick > with os.path -- stupid but familiar and doesn't put any runtime py lib > requirements on code -- or Orendorf's path module -- a little crufty, > but I really like the string subclass, because people can ignore its > specialness if they feel more comfortable with os.path -- or py.path, > which I'll admit is probably a more elegant interface. One way or the > other, I feel like I have to *accept* string paths as arguments, and > path objects seem less compelling when I'm always instantiating new ones > and whatnot. Hrm. Not that I don't hate os.path, mind you. I can understand all your reasonings here. Let me note a few things: - py.path.local(py.path.local(X)) == py.path.local(X) which means that you safely do: def f(fn): fn = py.path.local(fn) without fearing a double-wrap. - str(path) gives you the plain string representation - if more people really want a string subclass it's not exactly hard to add it i guess So I consider "adds a runtime dependency" the heaviest argument. Btw, the py lib did get larger with time (and will continue to do so) but i still hold that the lib is far from becoming a framework because almost all of its parts are usable on their own without knowing about the others parts. cheers, holger From arigo at tunes.org Thu Apr 7 13:29:40 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 7 Apr 2005 12:29:40 +0100 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050405213411.GM23818@solar.trillke.net> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> Message-ID: <20050407112940.GA16477@vicky.ecs.soton.ac.uk> Hi Holger, On Tue, Apr 05, 2005 at 11:34:11PM +0200, holger krekel wrote: > Not quite, instead of writing > > def __init__(self): > ... > > you just write > > def setup_method(self, method): > ... I think the point of Roy was that in the second case the 'self' object contains stuff left over by previous tests, which -- if you're not careful -- might interfere with this new test. In general however what kind of automatic instantiation should occur for TestXxx classes is just a matter of convention. Py's is not Unittest's and both make some sense... Armin From arigo at tunes.org Thu Apr 7 13:32:41 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 7 Apr 2005 12:32:41 +0100 Subject: [py-dev] execnet hangs Message-ID: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> Hi, I'm having a hard time debugging the following example, which just hangs on the very last receive(). On the remote site, the CHANNEL_DATA message from the last send() shows up in the debug log, but it doesn't get passed to my 'for' loop... Armin -+- import py code = py.code.Source( """ for name, args in channel: channel.send(1) """) gw = py.execnet.SshGateway('codespeak.net') def do(c, name, *args): c.send((name, args)) obj = c.receive() print obj return obj c1 = gw.remote_exec(code) do(c1, 'stat', '/.') c2 = gw.remote_exec(code) do(c2, 'readdir', '/') c3 = gw.remote_exec(code) do(c3, 'readdir', '/home') From roy at panix.com Thu Apr 7 14:56:26 2005 From: roy at panix.com (Roy Smith) Date: Thu, 7 Apr 2005 08:56:26 -0400 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050407112940.GA16477@vicky.ecs.soton.ac.uk> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> Message-ID: On Apr 7, 2005, at 7:29 AM, Armin Rigo wrote: > Hi Holger, > > On Tue, Apr 05, 2005 at 11:34:11PM +0200, holger krekel wrote: >> Not quite, instead of writing >> >> def __init__(self): >> ... >> >> you just write >> >> def setup_method(self, method): >> ... > > I think the point of Roy was that in the second case the 'self' object > contains stuff left over by previous tests, which -- if you're not > careful -- > might interfere with this new test. Yes, that was exactly my point. Although, after thinking about it more, I came to the realization that there's two different classes we're talking about. There's the test driver class, and there's the class you're testing. When I wrote my original note, I was blurring the two. In unittest, you get a new instances of TestDriver automatically, and it's up to you to create your own instance of ClassBeingTested in TestDriver.SetUp(). In py.test, you get a hand-me-down instance of TestDriver, and it's still up to you to create (or not, I suppose) your own instance of ClassBeingTested in TestDriver.setup_method(). So, given that new-found understanding, I think I'm OK with what py.test is doing. Py.test's re-use of the TestDriver instance gives you the opportunity to store state between individual test cases, should you desire to do so. Although, to be fair, you could do the same in unitest, by stashing the state in a global. > > In general however what kind of automatic instantiation should occur > for > TestXxx classes is just a matter of convention. Py's is not > Unittest's and > both make some sense... > > > Armin From pierre.barbier at cirad.fr Thu Apr 7 17:44:42 2005 From: pierre.barbier at cirad.fr (Pierre Barbier de Reuille) Date: Thu, 07 Apr 2005 17:44:42 +0200 Subject: [py-dev] Error with py.test and scipy ... Message-ID: <4255556A.8010507@cirad.fr> Hello, I just tried to use py.test. However, I experience segmentation fault when importing scipy module. Did anyone have the same problem before ? Is there a way to avoid it ? Thanks, Pierre -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68 From pierre.barbier at cirad.fr Thu Apr 7 17:46:13 2005 From: pierre.barbier at cirad.fr (Pierre Barbier de Reuille) Date: Thu, 07 Apr 2005 17:46:13 +0200 Subject: [py-dev] Error with py.test and scipy ... In-Reply-To: <4255556A.8010507@cirad.fr> References: <4255556A.8010507@cirad.fr> Message-ID: <425555C5.6070401@cirad.fr> oops ... I may say I'm using the latest version of py.test (checked out today), scipy 0.3.2 and python 2.3.5 on a linux debian plateform. Thanks ! Pierre Barbier de Reuille a ?crit : > Hello, > > I just tried to use py.test. However, I experience segmentation fault > when importing scipy module. Did anyone have the same problem before ? > Is there a way to avoid it ? > > Thanks, > > Pierre > -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68 From pierre.barbier at cirad.fr Thu Apr 7 18:40:04 2005 From: pierre.barbier at cirad.fr (Pierre Barbier de Reuille) Date: Thu, 07 Apr 2005 18:40:04 +0200 Subject: [py-dev] Error with py.test and scipy ... In-Reply-To: <4255556A.8010507@cirad.fr> References: <4255556A.8010507@cirad.fr> Message-ID: <42556264.4070500@cirad.fr> oops ... I may say I'm using the latest version of py.test (checked out today), scipy 0.3.2 and python 2.3.5 on a linux debian plateform. Thanks ! Pierre Barbier de Reuille a ?crit : > Hello, > > I just tried to use py.test. However, I experience segmentation fault > when importing scipy module. Did anyone have the same problem before ? > Is there a way to avoid it ? > > Thanks, > > Pierre > -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68 From arigo at tunes.org Thu Apr 7 20:33:28 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 7 Apr 2005 19:33:28 +0100 Subject: [py-dev] execnet hangs In-Reply-To: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> Message-ID: <20050407183328.GA15962@vicky.ecs.soton.ac.uk> Hi, On Thu, Apr 07, 2005 at 12:32:41PM +0100, Armin Rigo wrote: > I'm having a hard time debugging the following example, which just hangs on > the very last receive(). On the remote site, the CHANNEL_DATA message from > the last send() shows up in the debug log, but it doesn't get passed to my > 'for' loop... I had forgotten that gateways only create two threads to execute the code snippets to pass around. This bites me in this kind of case, where I want more than two loops waiting on the server (but still not a large number of them, overwise a single-loop channel-based approach would be preferred). Armin From jan at balster.info Thu Apr 7 21:50:18 2005 From: jan at balster.info (Jan Balster) Date: Thu, 07 Apr 2005 21:50:18 +0200 Subject: [py-dev] Error with py.test and scipy ... Message-ID: <42558EFA.9060205@balster.info> Pierre Barbier de Reuille wrote: > oops ... I may say I'm using the latest version of py.test (checked out > today), scipy 0.3.2 and python 2.3.5 on a linux debian plateform. > > Thanks ! > > Pierre Barbier de Reuille a ?crit : > >> Hello, >> >> I just tried to use py.test. However, I experience segmentation fault >> when importing scipy module. Did anyone have the same problem before ? Yes. I used a fresh compiled Scipy 0.3.2 and Python 2.4 with latest py.test on SuSE 9.2 linux. Example: class TestClass: def test_scipy(self): import scipy assert 1==1 Report: =============================================================================== scipy_________test_pytest.py[1] segmentation fault It seems to be a problem in scipy_base. Here is what i did: in scipy_base/__init__.py comment out: from index_tricks import * from function_base import * from polynomial import * from scimath import * in common.py remove amin, amax, extract, insert, poly1d, r_ from the "from scipy_base import" statement in pilutil.py remove amin, amax, mgrid from the "from scipy_base import" statement Now the little testcase runs without a segfault. Hope this works for you. Jan From hpk at trillke.net Thu Apr 7 23:17:15 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 7 Apr 2005 23:17:15 +0200 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050407112940.GA16477@vicky.ecs.soton.ac.uk> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> Message-ID: <20050407211715.GG23818@solar.trillke.net> Hi Armin, hi Roy, On Thu, Apr 07, 2005 at 12:29 +0100, Armin Rigo wrote: > On Tue, Apr 05, 2005 at 11:34:11PM +0200, holger krekel wrote: > > Not quite, instead of writing > > > > def __init__(self): > > ... > > > > you just write > > > > def setup_method(self, method): > > ... > > I think the point of Roy was that in the second case the 'self' object > contains stuff left over by previous tests, which -- if you're not careful -- > might interfere with this new test. ah, i missed this detail. Probably because such a situation requires buggy code, no? Schematically you would have: def test_1(self): self.x = 3 ... def test_2(self): ... use self.x ... which doesn't sound quite right. It can be argued that py.test should be more helpful by instantiating the instance on every test method. However, it is possible, as Roy mentions in his reply, that test_2 _wants_ to use self.x which would allow incremental testing. Howering-the-however, such incremental testing is fragile with respect to "--session" mode which reruns only failing tests. There is no sane way (short of running PyPy's typeinferencing or something :-) to detect that test_2 uses state that was created by test_1. Ok, ending these complicated considerations, i am 50:50 in instantiating the TestClass on every test method invocation. Anybody wants to weigh in on this detail? Hum, would it make sense to be able to explicitely specify incremental testing per-module or per-class? Sometimes I want to order tests by complexity and don't want to proceed if something fundamental fails early. cheers, holger From hpk at trillke.net Thu Apr 7 23:38:31 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 7 Apr 2005 23:38:31 +0200 Subject: [py-dev] Error with py.test and scipy ... In-Reply-To: <42558EFA.9060205@balster.info> References: <42558EFA.9060205@balster.info> Message-ID: <20050407213831.GI23818@solar.trillke.net> On Thu, Apr 07, 2005 at 21:50 +0200, Jan Balster wrote: > Pierre Barbier de Reuille wrote: > > oops ... I may say I'm using the latest version of py.test (checked out > > today), scipy 0.3.2 and python 2.3.5 on a linux debian plateform. > > > > Thanks ! > > > > Pierre Barbier de Reuille a ?crit : > > > >> Hello, > >> > >> I just tried to use py.test. However, I experience segmentation fault > >> when importing scipy module. Did anyone have the same problem before ? > > Yes. I used a fresh compiled Scipy 0.3.2 and Python 2.4 with latest > py.test on SuSE 9.2 linux. > > Example: > > class TestClass: > > def test_scipy(self): > import scipy > assert 1==1 > > Report: > =============================================================================== > scipy_________test_pytest.py[1] segmentation fault but a plain "import scipy" on python-2.4 works? > It seems to be a problem in scipy_base. > Here is what i did: > > in scipy_base/__init__.py comment out: > from index_tricks import * > from function_base import * > from polynomial import * > from scimath import * > > in common.py remove amin, amax, extract, insert, poly1d, r_ from the > "from scipy_base import" statement > > in pilutil.py remove amin, amax, mgrid from the "from scipy_base import" > statement > > Now the little testcase runs without a segfault. > Hope this works for you. if "import scipy" works then i'd like to find out the real cause for the problem with py.test interactions here. py.test is supposed to be non-intrusive ... and having to modify scipy in any way does not sound right. (we already had such a case with twisted's nevow, btw, and it was related to *cough*importing*cough*. cheers, holger From hpk at trillke.net Thu Apr 7 23:45:33 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 7 Apr 2005 23:45:33 +0200 Subject: [py-dev] execnet hangs In-Reply-To: <20050407183328.GA15962@vicky.ecs.soton.ac.uk> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> <20050407183328.GA15962@vicky.ecs.soton.ac.uk> Message-ID: <20050407214533.GJ23818@solar.trillke.net> On Thu, Apr 07, 2005 at 19:33 +0100, Armin Rigo wrote: > Hi, > > On Thu, Apr 07, 2005 at 12:32:41PM +0100, Armin Rigo wrote: > > I'm having a hard time debugging the following example, which just hangs on > > the very last receive(). On the remote site, the CHANNEL_DATA message from > > the last send() shows up in the debug log, but it doesn't get passed to my > > 'for' loop... > > I had forgotten that gateways only create two threads to execute the code > snippets to pass around. This bites me in this kind of case, where I want > more than two loops waiting on the server (but still not a large number of > them, overwise a single-loop channel-based approach would be preferred). Actually, as it happens, i was just wondering about this "2 threads" issue today (but wasn't online to my excuse :-). Of course, we could just increment this constant but this imposes setup/teardown penalties and generally costs resources. I'd rather think about an adaptive scheme where we allow to specify "maxthreads" and default it to 10 or so. Would that make sense to you? Luckily such things are relatively easy to implement in Python, i've done similar things with ACE/CORBA in C++ in a previous life ... SEGFAULT. cheers, holger From rkern at ucsd.edu Thu Apr 7 23:54:56 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu, 07 Apr 2005 14:54:56 -0700 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <20050407213831.GI23818@solar.trillke.net> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> Message-ID: holger krekel wrote: > if "import scipy" works then i'd like to find out the real > cause for the problem with py.test interactions here. > py.test is supposed to be non-intrusive ... and having > to modify scipy in any way does not sound right. > (we already had such a case with twisted's nevow, btw, > and it was related to *cough*importing*cough*. Scipy uses a lazy importing scheme implemented in scipy_base.ppimport. I'll try to get around to testing it myself. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hpk at trillke.net Fri Apr 8 00:39:02 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 00:39:02 +0200 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> Message-ID: <20050407223902.GL23818@solar.trillke.net> On Thu, Apr 07, 2005 at 14:54 -0700, Robert Kern wrote: > holger krekel wrote: > > >if "import scipy" works then i'd like to find out the real > >cause for the problem with py.test interactions here. > >py.test is supposed to be non-intrusive ... and having > >to modify scipy in any way does not sound right. > >(we already had such a case with twisted's nevow, btw, > >and it was related to *cough*importing*cough*. > > Scipy uses a lazy importing scheme implemented in scipy_base.ppimport. import magic's interfering with each other ... this had to happen sooner or later. And we've had Valentino having problems with twisted's nevow already which uses crazy import hacks ... welcome in scary traceback land. This strongly suggests to me that py.test will have to try harder to reduce its import magic. Luckily, i have an alternative approach which removes the need to have a custom import hook for the py lib. I am checking that into the collect-branch (which will soon get merged, didn't i mention that last week?) On a side note, i can't remember the number of times anymore that i - often in agreemeent with others - have stumbled on this pile of hacks called the import system. What especially bugs me is that i am absolutely sure that Python could have (and would deserve!) a really nice thought-through import system. Hum, i guess that if a pile of hacks passes a certain complexity level then it has strong tendencies to keep piling. cheers, holger From roy at panix.com Fri Apr 8 00:46:08 2005 From: roy at panix.com (Roy Smith) Date: Thu, 7 Apr 2005 18:46:08 -0400 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050407211715.GG23818@solar.trillke.net> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> Message-ID: <1e199c56071ad7720ab53ce17a3feefb@panix.com> On Apr 7, 2005, at 5:17 PM, holger krekel wrote: > > Howering-the-however, such incremental testing is fragile with respect > to "--session" mode which reruns only failing tests. Is --session documented anywhere? I couldn't find it. > Ok, ending these complicated considerations, i am 50:50 in > instantiating the TestClass on every test method invocation. > Anybody wants to weigh in on this detail? > I like the idea that each test case is completely independent of every other test case. In other words, I prefer a new TestClass instance for each test case. This will also make it more like unittest (it's fine to innovate when there's a significant improvement, but change for the sake of change is less attractive). I also don't see the need for all of setup_module, setup_class, and setup_method. I only really see real setup scenarios -- stuff that should be done once, and stuff that should be done for every test case. Having three hooks for two scenarios seems like more complexity than is required. From pobrien at orbtech.com Fri Apr 8 00:45:54 2005 From: pobrien at orbtech.com (Patrick K. O'Brien) Date: Thu, 07 Apr 2005 17:45:54 -0500 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <20050407223902.GL23818@solar.trillke.net> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> <20050407223902.GL23818@solar.trillke.net> Message-ID: <4255B822.9010401@orbtech.com> holger krekel wrote: > On Thu, Apr 07, 2005 at 14:54 -0700, Robert Kern wrote: >> >>Scipy uses a lazy importing scheme implemented in scipy_base.ppimport. > > > import magic's interfering with each other ... this had to happen > sooner or later. And we've had Valentino having problems with > twisted's nevow already which uses crazy import hacks ... welcome > in scary traceback land. This strongly suggests to me that py.test > will have to try harder to reduce its import magic. Luckily, i have > an alternative approach which removes the need to have a custom > import hook for the py lib. I am checking that into the > collect-branch (which will soon get merged, didn't i mention that > last week?) Count Schevo as yet another project that does its share of import magic. We've managed to keep our magic compatible with py.test's magic and nevow's magic, and I'd hate to lose that. -- Patrick K. O'Brien Orbtech http://www.orbtech.com Schevo http://www.schevo.org Pypersyst http://www.pypersyst.org From huymail at swiftdsl.com.au Fri Apr 8 01:00:12 2005 From: huymail at swiftdsl.com.au (Huy) Date: Fri, 08 Apr 2005 09:00:12 +1000 Subject: [py-dev] unitest.main() equivalent Message-ID: <4255BB7C.9030300@swiftdsl.com.au> Hi, I was wondering if there is a unittest.main() equivalent in py.test I'm basically trying to run my test directly in eclipse using pydev. Thanks, Huy From hpk at trillke.net Fri Apr 8 01:18:21 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 01:18:21 +0200 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <1e199c56071ad7720ab53ce17a3feefb@panix.com> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> Message-ID: <20050407231821.GM23818@solar.trillke.net> Hi Roy, On Thu, Apr 07, 2005 at 18:46 -0400, Roy Smith wrote: > On Apr 7, 2005, at 5:17 PM, holger krekel wrote: > > > >Howering-the-however, such incremental testing is fragile with respect > >to "--session" mode which reruns only failing tests. > > Is --session documented anywhere? I couldn't find it. > > >Ok, ending these complicated considerations, i am 50:50 in > >instantiating the TestClass on every test method invocation. > >Anybody wants to weigh in on this detail? > > I like the idea that each test case is completely independent of every > other test case. In other words, I prefer a new TestClass instance for > each test case. This will also make it more like unittest (it's fine > to innovate when there's a significant improvement, but change for the > sake of change is less attractive). ok, but i am still interested in more opinions. The thing is that i currently think that it could make things slightly less nice in the implementation. The current collect-py branch has rather general semantics for collecting objects. As an illustration, this is the output from "py.test --collectonly test_util.py" (in the branch): With instantiating per method this would become: ... Ugly, isn't it? :^) > I also don't see the need for all of setup_module, setup_class, and > setup_method. I only really see real setup scenarios -- stuff that > should be done once, and stuff that should be done for every test case. "should be done once" is a matter of defining the specific scope. per-modules, per-class and per-method seems like a natural choice to me. In addition i would bet that if we didn't have setup_method then people would say "but unittest has setUp and tearDown! where is it in py.test?", clearly at least a marketing catastrophe :-) cheers, holger From hpk at trillke.net Fri Apr 8 01:21:57 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 01:21:57 +0200 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <4255B822.9010401@orbtech.com> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> <20050407223902.GL23818@solar.trillke.net> <4255B822.9010401@orbtech.com> Message-ID: <20050407232157.GN23818@solar.trillke.net> On Thu, Apr 07, 2005 at 17:45 -0500, Patrick K. O'Brien wrote: > holger krekel wrote: > > On Thu, Apr 07, 2005 at 14:54 -0700, Robert Kern wrote: > >> > >>Scipy uses a lazy importing scheme implemented in scipy_base.ppimport. > > > > > > import magic's interfering with each other ... this had to happen > > sooner or later. And we've had Valentino having problems with > > twisted's nevow already which uses crazy import hacks ... welcome > > in scary traceback land. This strongly suggests to me that py.test > > will have to try harder to reduce its import magic. Luckily, i have > > an alternative approach which removes the need to have a custom > > import hook for the py lib. I am checking that into the > > collect-branch (which will soon get merged, didn't i mention that > > last week?) > > Count Schevo as yet another project that does its share of import magic. > We've managed to keep our magic compatible with py.test's magic and > nevow's magic, and I'd hate to lose that. I would be surprised if my brand-new hack breaks yours (because that is exactly what it is trying to avoid) but i've learned that "import" makes the impossible possible and vice versa. cheers, holger From hpk at trillke.net Fri Apr 8 01:28:56 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 01:28:56 +0200 Subject: [py-dev] unitest.main() equivalent In-Reply-To: <4255BB7C.9030300@swiftdsl.com.au> References: <4255BB7C.9030300@swiftdsl.com.au> Message-ID: <20050407232856.GO23818@solar.trillke.net> Hi Huy, On Fri, Apr 08, 2005 at 09:00 +1000, Huy wrote: > I was wondering if there is a unittest.main() equivalent in py.test not really. > I'm basically trying to run my test directly in eclipse using pydev. 'pydev' means py.test, i guess? Is it so that one runs tests with eclipse by running the file directly? Isn't there a way to customize eclipse by specifying that "python path/to/py.test FILE_OR_DIRECTORY" should be run for testing? cheers, holger From dialtone at divmod.com Fri Apr 8 01:35:11 2005 From: dialtone at divmod.com (Valentino Volonghi aka Dialtone) Date: Fri, 08 Apr 2005 01:35:11 +0200 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <20050407232157.GN23818@solar.trillke.net> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> <20050407223902.GL23818@solar.trillke.net> <4255B822.9010401@orbtech.com> <20050407232157.GN23818@solar.trillke.net> Message-ID: <4255C3AF.2040904@divmod.com> holger krekel wrote: >I would be surprised if my brand-new hack breaks yours (because that >is exactly what it is trying to avoid) but i've learned that "import" >makes the impossible possible and vice versa. > >cheers, > > It was a quick fix in my case :). It shouldn't be hard to figure out a common failing pattern in import magic. for example: nevow was failing because of namedAny that imports a module from a string representing the entire path in PYTHONPATH. from nevow import util In [6]:util.namedAny('nevow.flat.ten') Out[6]: namedAny is this one: def namedAny(name): """Get a fully named package, module, module-global object, or attribute. """ names = name.split('.') topLevelPackage = None moduleNames = names[:] while not topLevelPackage: try: trialname = '.'.join(moduleNames) topLevelPackage = __import__(trialname) except ImportError: # if the ImportError happened in the module being imported, # this is a failure that should be handed to our caller. # count stack frames to tell the difference. # string-matching is another technique, but I think it could be # fooled in some funny cases #if sys.exc_info()[1] != "cannot import name %s" % trialname: # raise import traceback if len(traceback.extract_tb(sys.exc_info()[2])) > 1: raise moduleNames.pop() obj = topLevelPackage for n in names[1:]: obj = getattr(obj, n) return obj But I can't remember exactly the failure. The ML is here to help (something about the PYTHONPATH anyway). -- Valentino Volonghi aka Dialtone Now Running MacOSX 10.3.8 Blog: http://vvolonghi.blogspot.com http://weever.berlios.de From ianb at colorstudy.com Fri Apr 8 02:12:40 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 07 Apr 2005 19:12:40 -0500 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050407231821.GM23818@solar.trillke.net> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> <20050407231821.GM23818@solar.trillke.net> Message-ID: <4255CC78.8010403@colorstudy.com> holger krekel wrote: >>I like the idea that each test case is completely independent of every >>other test case. In other words, I prefer a new TestClass instance for >>each test case. This will also make it more like unittest (it's fine >>to innovate when there's a significant improvement, but change for the >>sake of change is less attractive). > > > ok, but i am still interested in more opinions. The thing is that > i currently think that it could make things slightly less nice > in the implementation. I think reinstantiating just makes sense, and is what people expect from unittest experience. Handling an instance also makes sense, but isn't exclusive of this (when you instantiate the class manually, passing in arguments -- something that is hard in unittest). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Apr 8 02:28:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 07 Apr 2005 19:28:39 -0500 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <20050407232157.GN23818@solar.trillke.net> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> <20050407223902.GL23818@solar.trillke.net> <4255B822.9010401@orbtech.com> <20050407232157.GN23818@solar.trillke.net> Message-ID: <4255D037.6030608@colorstudy.com> holger krekel wrote: > On Thu, Apr 07, 2005 at 17:45 -0500, Patrick K. O'Brien wrote: > >>holger krekel wrote: >> >>>On Thu, Apr 07, 2005 at 14:54 -0700, Robert Kern wrote: >>> >>>>Scipy uses a lazy importing scheme implemented in scipy_base.ppimport. >>> >>> >>>import magic's interfering with each other ... this had to happen >>>sooner or later. And we've had Valentino having problems with >>>twisted's nevow already which uses crazy import hacks ... welcome >>>in scary traceback land. This strongly suggests to me that py.test >>>will have to try harder to reduce its import magic. Luckily, i have >>>an alternative approach which removes the need to have a custom >>>import hook for the py lib. I am checking that into the >>>collect-branch (which will soon get merged, didn't i mention that >>>last week?) >> >>Count Schevo as yet another project that does its share of import magic. >> We've managed to keep our magic compatible with py.test's magic and >>nevow's magic, and I'd hate to lose that. > > > I would be surprised if my brand-new hack breaks yours (because that > is exactly what it is trying to avoid) but i've learned that "import" > makes the impossible possible and vice versa. There sure are a lot of people doing import hacks these days -- as I look around I keep encountering more of them. I guess it takes a while for people to become comfortable with such magic, but people are clearly feeling comfortable now. I am guessing that this could all have bad interactions with eggs, and who knows what else. I can barely manage right now to bring together multiple packages easily with *normal* import semantics (stupid sys.path), and I don't know how this stuff will come together in the future. I wonder if people are overloading the idea of "module" when really they want to create entirely new kinds of objects-built-from-Python-files that aren't modules at all. Do we need a new metaphor? Do we need one Magic But Really Well Spec'ed import hook that will satisfy everyone's needs? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From dripton at ripton.net Fri Apr 8 04:40:44 2005 From: dripton at ripton.net (David Ripton) Date: Thu, 7 Apr 2005 19:40:44 -0700 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050407231821.GM23818@solar.trillke.net> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> <20050407231821.GM23818@solar.trillke.net> Message-ID: <20050408024044.GA1576@vidar.dreamhost.com> On 2005.04.08 01:18:21 +0000, holger krekel wrote: > > I like the idea that each test case is completely independent of every > > other test case. In other words, I prefer a new TestClass instance for > > each test case. This will also make it more like unittest (it's fine > > to innovate when there's a significant improvement, but change for the > > sake of change is less attractive). > > ok, but i am still interested in more opinions. The thing is that Re-initializing by default feels cleaner to me. It's what xUnit users expect. Though of course it needs to be possible to override this and explicitly reuse something expensive like a database connection. > > I also don't see the need for all of setup_module, setup_class, and > > setup_method. I only really see real setup scenarios -- stuff that > > should be done once, and stuff that should be done for every test case. > > "should be done once" is a matter of defining the specific scope. > per-modules, per-class and per-method seems like a natural choice > to me. In addition i would bet that if we didn't have setup_method > then people would say "but unittest has setUp and tearDown! where is it > in py.test?", clearly at least a marketing catastrophe :-) I think all three are useful, even though it's unlikely that the same module would use all three. They're simple and orthogonal enough that I think this is useful variety rather than feeping creaturism. setup_module is useful for reusing database connections across a multi-class test. Also for when you have expensive test initialization but no test classes because you've chosen to use top-level test functions instead. setup_class lets you put multiple test classes that have different expensive initialization in one file, rather than making you split them up into separate files to use setup_module. setup_method is necessary because unittest has it. The ability to translate existing unittest modules to py.test with no thinking required is key if you want unittest users to switch their existing projects, rather than waiting until their next project to switch. (Ideally, there shouldn't be any grunt work either -- a unittest_to_pytest script would be nice to have, even if it wasn't 100% perfect.) -- David Ripton dripton at ripton.net From roy at panix.com Fri Apr 8 04:49:00 2005 From: roy at panix.com (Roy Smith) Date: Thu, 7 Apr 2005 22:49:00 -0400 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050408024044.GA1576@vidar.dreamhost.com> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> <20050407231821.GM23818@solar.trillke.net> <20050408024044.GA1576@vidar.dreamhost.com> Message-ID: <3b1cfa36c62ecdeb42306127753b5f91@panix.com> On Apr 7, 2005, at 10:40 PM, David Ripton wrote: > setup_class lets you put multiple test classes that have different > expensive initialization in one file, rather than making you split them > up into separate files to use setup_module. Fair enough. I was thinking one test class per file/module, but you're absolutely right that you might have multiple classes in one module. And then having distinct setup_module() and setup_class() makes perfect sense. From hpk at trillke.net Fri Apr 8 05:10:28 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 05:10:28 +0200 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <1e199c56071ad7720ab53ce17a3feefb@panix.com> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> Message-ID: <20050408031028.GQ23818@solar.trillke.net> On Thu, Apr 07, 2005 at 18:46 -0400, Roy Smith wrote: > On Apr 7, 2005, at 5:17 PM, holger krekel wrote: > > > >Howering-the-however, such incremental testing is fragile with respect > >to "--session" mode which reruns only failing tests. > > Is --session documented anywhere? I couldn't find it. Ups, i overread that earlier. Actually i would have sworn that there was documentation about '--session' but apparently i was dreaming. Maybe the name should be changed to something like '--loop' which hopefully makes it more obvious that py.test will continously run all your tests, restarting every time a python file in your project changes. On each loop run it records failing tests and reruns only those. If there are no failing tests left, it starts all over. cheers, holger From mwh at python.net Fri Apr 8 10:20:42 2005 From: mwh at python.net (Michael Hudson) Date: Fri, 08 Apr 2005 09:20:42 +0100 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050408024044.GA1576@vidar.dreamhost.com> (David Ripton's message of "Thu, 7 Apr 2005 19:40:44 -0700") References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> <20050407231821.GM23818@solar.trillke.net> <20050408024044.GA1576@vidar.dreamhost.com> Message-ID: <2mzmw97k85.fsf@starship.python.net> David Ripton writes: > setup_method is necessary because unittest has it. The ability to > translate existing unittest modules to py.test with no thinking required > is key if you want unittest users to switch their existing projects, > rather than waiting until their next project to switch. (Ideally, there > shouldn't be any grunt work either -- a unittest_to_pytest script would > be nice to have, even if it wasn't 100% perfect.) Err, we have one of these already, for PyPy. I don't know if it's generally useful, though. Cheers, mwh -- I don't remember any dirty green trousers. -- Ian Jackson, ucam.chat From arigo at tunes.org Fri Apr 8 10:48:29 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Apr 2005 09:48:29 +0100 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <2mzmw97k85.fsf@starship.python.net> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> <20050407231821.GM23818@solar.trillke.net> <20050408024044.GA1576@vidar.dreamhost.com> <2mzmw97k85.fsf@starship.python.net> Message-ID: <20050408084829.GA31081@vicky.ecs.soton.ac.uk> Hi, On Fri, Apr 08, 2005 at 09:20:42AM +0100, Michael Hudson wrote: > > shouldn't be any grunt work either -- a unittest_to_pytest script would > > be nice to have, even if it wasn't 100% perfect.) > > Err, we have one of these already, for PyPy. I don't know if it's > generally useful, though. http://codespeak.net/svn/pypy/dist/pypy/tool/utestconvert.py I think that apart from 5 lines for extra conversion cases specific to PyPy, the tool is pretty much general and should be available somewhere with the py lib. Armin From arigo at tunes.org Fri Apr 8 10:52:33 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Apr 2005 09:52:33 +0100 Subject: [py-dev] execnet hangs In-Reply-To: <20050407214533.GJ23818@solar.trillke.net> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> <20050407183328.GA15962@vicky.ecs.soton.ac.uk> <20050407214533.GJ23818@solar.trillke.net> Message-ID: <20050408085233.GB31081@vicky.ecs.soton.ac.uk> Hi Holger, On Thu, Apr 07, 2005 at 11:45:33PM +0200, holger krekel wrote: > rather think about an adaptive scheme where we allow to > specify "maxthreads" and default it to 10 or so. Would that > make sense to you? Not really. Anything that, like my example above, appears to work until some small number (2 or 10) and completely deadlocks above that number is bound to create surprises and bugs. Why not create new 'executor' threads when none is available, with no upper bound? I'd much prefer buggy programs to trash the system with resource consumption than nicely written programs deadlock because of an internal limit. It should be clear that remote_exec() creates a new thread on the remote side, and that if you are going to do really a lot of remove_exec() you should do only one and use channels. Armin From hpk at trillke.net Fri Apr 8 12:41:16 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 12:41:16 +0200 Subject: [py-dev] execnet hangs In-Reply-To: <20050408085233.GB31081@vicky.ecs.soton.ac.uk> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> <20050407183328.GA15962@vicky.ecs.soton.ac.uk> <20050407214533.GJ23818@solar.trillke.net> <20050408085233.GB31081@vicky.ecs.soton.ac.uk> Message-ID: <20050408104116.GS23818@solar.trillke.net> Hi Armin, On Fri, Apr 08, 2005 at 09:52 +0100, Armin Rigo wrote: > On Thu, Apr 07, 2005 at 11:45:33PM +0200, holger krekel wrote: > > rather think about an adaptive scheme where we allow to > > specify "maxthreads" and default it to 10 or so. Would that > > make sense to you? > > Not really. Anything that, like my example above, appears to work until some > small number (2 or 10) and completely deadlocks above that number is bound to > create surprises and bugs. Full ack. However, nobody forces us to deadlock if we are exceeding the number of maximum threads. > Why not create new 'executor' threads when none is available, with no upper > bound? I'd much prefer buggy programs to trash the system with resource > consumption than nicely written programs deadlock because of an internal > limit. Not so sure about this. I want to do things on production machines (codespeak.net among others) with execnet and i wouldn't want to starve resources there because of a casual code bug :-) > It should be clear that remote_exec() creates a new thread on the > remote side, and that if you are going to do really a lot of remove_exec() you > should do only one and use channels. Using channels and doing your own multiplexing is painful. I'd say we should have a default of 10 for maxthreads but allow it to be None, meaning no bounds. If the bound is exceeded a Gateway.ThreadsExhausted (subclass of RuntimeError) would be raised on remote_exec() side. The point is that it is reasonably hard for a user of py.execnet to restrict the number of threads themselves (would probably invole subclassing etc.pp.). Is that agreeable? cheers, holger From arigo at tunes.org Fri Apr 8 12:52:43 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Apr 2005 11:52:43 +0100 Subject: [py-dev] execnet hangs In-Reply-To: <20050408104116.GS23818@solar.trillke.net> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> <20050407183328.GA15962@vicky.ecs.soton.ac.uk> <20050407214533.GJ23818@solar.trillke.net> <20050408085233.GB31081@vicky.ecs.soton.ac.uk> <20050408104116.GS23818@solar.trillke.net> Message-ID: <20050408105243.GB25009@vicky.ecs.soton.ac.uk> Hi Holger, On Fri, Apr 08, 2005 at 12:41:16PM +0200, holger krekel wrote: > Full ack. However, nobody forces us to deadlock if we are > exceeding the number of maximum threads. My point is that perfectly reasonable examples cause deadlocks because of this "feature". Detecting the problem and throwing an exception is not a good solution either. It causes nice small programs to start crashing when you scale them up. I don't see why we can't just document: "remote_exec() creates a new thread on the remote side", and be done with it. A program needs to seriously screw up to produce a ton of calls to remote_exec() and starve the resources; it is just the same as a program that, by error, calls thread.start_new_thread() far too many times. It could occur, but I've never seen that problem in practice and I'm definitely on the opinion that any default bound like 10 -- even if you can disable it -- is going to cause bugs. (And 10 is worse than 2 because you need to scale more before you see the bug.) I don't mind if you want to enforce a bound in some situations but the default should be no bound. Armin From hpk at trillke.net Fri Apr 8 13:31:39 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 13:31:39 +0200 Subject: [py-dev] Re: cmdline options / rev 10344 In-Reply-To: <20050408082547.GB7077@vicky.ecs.soton.ac.uk> References: <20050405173828.GI23818@solar.trillke.net> <20050408082547.GB7077@vicky.ecs.soton.ac.uk> Message-ID: <20050408113139.GU23818@solar.trillke.net> Hi Armin, i CCed to py-dev in case others are interested (beware everyone, we are touching some deeper implementation issues here). The following discussion starts off at the py-collect refactoring of cmdline option handling with respect to projects having custom command line options for py.test. On Fri, Apr 08, 2005 at 09:25 +0100, Armin Rigo wrote: > On Tue, Apr 05, 2005 at 07:38:28PM +0200, holger krekel wrote: > > if you feel like it it would be great if you would > > comment on rev 10344 on the py.test branch (i took > > care to write a hopefully understandable log message). > > Well, this problem is messy (did we mention that already?) and I think you > have found a quite reasonable solution. The "no-API" paradigm doesn't really > apply around conftest files anyway: if I look at PyPy's, they depend on quite > a lot of internal py.test stuff. Having to actually call back a py.test.xyz() > function doesn't change much. Except that it is more obvious how you can get at the resulting values of custom Option definitions. The idea is to try to reduce the globalness of the py.test API (and py.test.option.SOMETHING is pretty global ...). > Maybe conftest doesn't have to know about the Option class? I mean, instead > of py.test.addoptions('group', Option(...), Option(...)) we could have > something like multiple calls to py.test.addoption('group', ...). Yes, well, i agree to the intention but they are practical problems. Single calls would make it non-obvious how to get to the "option" value container from the conftest.py side. And passing a list of tuples or something would deride the use of named arguments (which is the only way to somehow maintain readable code). Unfortuantely, I didn't find out a way out of this mind-looping considerations so far. > I guess I'll review the rest of the py-collect branch sometime soon... Here are a few particularly interesting areas: - there is a file called "py/documentation/releasescheme.txt" which describes the planned approach towards releasing the py lib. I'd like PyPy to follow the same conventions so it makes double sense if you take a critical look. - regarding reducing import mess: py.test now uses py.path.local().pyimport() with a previously discussed algorithm of constructing a package name and consulting the root package's __package__.importfrompath() to get to a canonical module import. I am thinking of somehow keeping getpymodule()'s functionality of loading c-modules on the fly but otherwise completly deleting all stuff related to custom import hooks. This derides us of having "direct remote imports" but i think that's kind of a hopeless mess and somewhat unreal anyway (we can move the custom import hook hacks to some private directory to be able to reinstate it more easily if need be). In effect, for execnet we probably want to settle on an (efficient) filetransfer protocol but otherwise just do local imports. Are we going to cry tears not having our own custom import hook anymore? I don't think so. - Collectors are Items are Collectors: they merged quite a bit and have uniform signatures as seen from the (previously named 'Driver') Session. The following properties are true for any collector or item (usually named 'colitem' in the code): assert hasattr(colitem, 'name') assert colitem.parent is None or isinstance(colitem.parent, Collector) res = colitem.run() assert isiterable(res) or res is None # if res is iterable the session recurses into it if isiterable(res): for x in res: assert isinstance(x, CollectorOrItem) # a colitem can be reconstructed from a root collector and a # list of names colitems = colitem.listchain() current = colitems.pop(0) while colitems: current = current.join(colitems.pop(0).name) assert current == colitem To see a concrete tree of collectors/items, you can issue:: py.test --collectonly which shows more output than on the current trunk. ok, that should be enough for starters :-) The idea is that after the merge of the branch i soon want to cut a py-0.8.0 release and beta-test the release process. Guaruantees regarding APIs will only become effective after a py-1.0.0 which i want to have before EuroPython. cheers, holger From arigo at tunes.org Fri Apr 8 13:29:44 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Apr 2005 12:29:44 +0100 Subject: [py-dev] execnet hangs In-Reply-To: <20050408104116.GS23818@solar.trillke.net> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> <20050407183328.GA15962@vicky.ecs.soton.ac.uk> <20050407214533.GJ23818@solar.trillke.net> <20050408085233.GB31081@vicky.ecs.soton.ac.uk> <20050408104116.GS23818@solar.trillke.net> Message-ID: <20050408112944.GA8163@vicky.ecs.soton.ac.uk> Hi Holger, Another solution if you want to avoid the problems of infinite thread creation would be to introduce a timeout-based approach: if all executor threads are busy and the next item on the executor queue doesn't get consumed for more than 0.05 seconds (for example), then a new executor thread is started. This would make the process interruptible before it consumes all resources, and it would also discourage people from using remote_exec() to start a large number of threads. Alternatively I could try to rewrite execnet to use greenlets internally :-) Armin From hpk at trillke.net Fri Apr 8 13:40:05 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 13:40:05 +0200 Subject: [py-dev] execnet hangs In-Reply-To: <20050408105243.GB25009@vicky.ecs.soton.ac.uk> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> <20050407183328.GA15962@vicky.ecs.soton.ac.uk> <20050407214533.GJ23818@solar.trillke.net> <20050408085233.GB31081@vicky.ecs.soton.ac.uk> <20050408104116.GS23818@solar.trillke.net> <20050408105243.GB25009@vicky.ecs.soton.ac.uk> Message-ID: <20050408114005.GV23818@solar.trillke.net> Hi Armin, On Fri, Apr 08, 2005 at 11:52 +0100, Armin Rigo wrote: > On Fri, Apr 08, 2005 at 12:41:16PM +0200, holger krekel wrote: > > Full ack. However, nobody forces us to deadlock if we are > > exceeding the number of maximum threads. > > My point is that perfectly reasonable examples cause deadlocks because of this > "feature". Detecting the problem and throwing an exception is not a good > solution either. It causes nice small programs to start crashing when you > scale them up. > > I don't see why we can't just document: "remote_exec() creates a new thread on > the remote side", and be done with it. A program needs to seriously screw up > to produce a ton of calls to remote_exec() and starve the resources; it is > just the same as a program that, by error, calls thread.start_new_thread() far > too many times. It could occur, but I've never seen that problem in practice > and I'm definitely on the opinion that any default bound like 10 -- even if > you can disable it -- is going to cause bugs. (And 10 is worse than 2 because > you need to scale more before you see the bug.) > > I don't mind if you want to enforce a bound in some situations but the default > should be no bound. Ok, i am fine with making maxthreads=None (meaning unbound) the default. But allowing to bound the number of threads makes sense as well (unless it is too hard to implement :-). Or could we actually detect the "can't create a thread anymore" case on the remote side and raise a ThreadsExhausted exception to the remote_exec() caller [*]? Seems to be the best of all worlds ... holger [*] which one would only notice when trying to channel.*() something because remote_exec() is fully asynchronous. It is helpful to get an exception instead of just blocking and not knowing why, isn't it? From hpk at trillke.net Fri Apr 8 13:44:54 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 13:44:54 +0200 Subject: [py-dev] execnet hangs In-Reply-To: <20050408112944.GA8163@vicky.ecs.soton.ac.uk> References: <20050407113241.GA24018@vicky.ecs.soton.ac.uk> <20050407183328.GA15962@vicky.ecs.soton.ac.uk> <20050407214533.GJ23818@solar.trillke.net> <20050408085233.GB31081@vicky.ecs.soton.ac.uk> <20050408104116.GS23818@solar.trillke.net> <20050408112944.GA8163@vicky.ecs.soton.ac.uk> Message-ID: <20050408114454.GW23818@solar.trillke.net> Hi Armin, On Fri, Apr 08, 2005 at 12:29 +0100, Armin Rigo wrote: > Hi Holger, > > Another solution if you want to avoid the problems of infinite thread creation > would be to introduce a timeout-based approach: if all executor threads are > busy and the next item on the executor queue doesn't get consumed for more > than 0.05 seconds (for example), then a new executor thread is started. This > would make the process interruptible before it consumes all resources, and it > would also discourage people from using remote_exec() to start a large number > of threads. > > Alternatively I could try to rewrite execnet to use greenlets internally :-) Append an "optionally" to that and that would be great :-) Hey, and the twisted "we hate threads" folks would herold you, i guess. If you want the ultimate challenge then consider allowing a mix of os-threads and greenlets to keep the thing responsive if one greenlet doesn't cooperate nicely :-) holger From p.f.moore at gmail.com Fri Apr 8 14:43:06 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 8 Apr 2005 13:43:06 +0100 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <4255D037.6030608@colorstudy.com> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> <20050407223902.GL23818@solar.trillke.net> <4255B822.9010401@orbtech.com> <20050407232157.GN23818@solar.trillke.net> <4255D037.6030608@colorstudy.com> Message-ID: <79990c6b0504080543364d0e35@mail.gmail.com> On Apr 8, 2005 1:28 AM, Ian Bicking wrote: > I wonder if people are overloading the idea of "module" when really they > want to create entirely new kinds of objects-built-from-Python-files > that aren't modules at all. Do we need a new metaphor? Do we need one > Magic But Really Well Spec'ed import hook that will satisfy everyone's > needs? PEP 302 was intended to unify the import hook landscape, and make it easier for hooks to work together. My impression is that it failed, but I'm not sure why. My name is on the PEP, but I really only did documentation, testing, and a little bit of design. Most of the good stuff was Just's. I'd like to think that "tidying up" PEP 302 would solve the issues, but I don't really have a good use case - my interest was with regard to zip imports, and the sort of areas that eggs are addressing, but I was looking forward to a situation where PEP 302 hooks were established, and there would be no need to cater for custom approaches. Sadly, I no longer see that day approaching :-( An immediate question springs to mind - what is the py magic doing, and how does PEP 302 fail to cater for its needs? (Repeat this question for any other package with nasty import magic). Paul. PS While I'm opening this can of worms again, I can't offer any assistance making things happen in the code, as I am no longer able to build/patch Python (I only have MSVC6). So any changes which seem needed will require a champion... From hpk at trillke.net Fri Apr 8 14:50:11 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 14:50:11 +0200 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <4255C3AF.2040904@divmod.com> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> <20050407223902.GL23818@solar.trillke.net> <4255B822.9010401@orbtech.com> <20050407232157.GN23818@solar.trillke.net> <4255C3AF.2040904@divmod.com> Message-ID: <20050408125011.GZ23818@solar.trillke.net> Hi Valentino, my reply went to pypy-dev because i screwed up CCs. Here is the same reply going to py-dev instead, please take care that you reply to this message (if you want to reply ...). On Fri, Apr 08, 2005 at 01:33 +0200, Valentino Volonghi aka Dialtone wrote: > holger krekel wrote: > > >I would be surprised if my brand-new hack breaks yours (because that > >is exactly what it is trying to avoid) but i've learned that "import" > >makes the impossible possible and vice versa. > > > >cheers, > > > > > It was a quick fix in my case :). Everybody says that, see "piles of hacks" earlier :-) > It shouldn't be hard to figure out a common failing pattern in import > magic. for example: > nevow was failing because of namedAny that imports a module from a > string representing the entire path in PYTHONPATH. brain shaking already ... > from nevow import util > In [6]:util.namedAny('nevow.flat.ten') > Out[6]: '/Volumes/dati/Sviluppo/Nevow/nevow/flat/ten.pyc'> > > namedAny is this one: > > def namedAny(name): > """Get a fully named package, module, module-global object, or > attribute. > """ > names = name.split('.') > topLevelPackage = None > moduleNames = names[:] > while not topLevelPackage: > try: > trialname = '.'.join(moduleNames) > topLevelPackage = __import__(trialname) > except ImportError: > # if the ImportError happened in the module being imported, > # this is a failure that should be handed to our caller. > # count stack frames to tell the difference. > > # string-matching is another technique, but I think it could be > # fooled in some funny cases > #if sys.exc_info()[1] != "cannot import name %s" % trialname: > # raise > import traceback > if len(traceback.extract_tb(sys.exc_info()[2])) > 1: > raise > moduleNames.pop() > > obj = topLevelPackage > for n in names[1:]: > obj = getattr(obj, n) > > return obj > > But I can't remember exactly the failure. The ML is here to help > (something about the PYTHONPATH anyway). I have written similar code and there were always side cases that made it at least hard to debug and find the actual problem in the program. cheers, holger From hpk at trillke.net Fri Apr 8 15:05:15 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 15:05:15 +0200 Subject: [py-dev] Re: Error with py.test and scipy ... In-Reply-To: <79990c6b0504080543364d0e35@mail.gmail.com> References: <42558EFA.9060205@balster.info> <20050407213831.GI23818@solar.trillke.net> <20050407223902.GL23818@solar.trillke.net> <4255B822.9010401@orbtech.com> <20050407232157.GN23818@solar.trillke.net> <4255D037.6030608@colorstudy.com> <79990c6b0504080543364d0e35@mail.gmail.com> Message-ID: <20050408130515.GA23818@solar.trillke.net> Hi Paul, On Fri, Apr 08, 2005 at 13:43 +0100, Paul Moore wrote: > On Apr 8, 2005 1:28 AM, Ian Bicking wrote: > > > I wonder if people are overloading the idea of "module" when really they > > want to create entirely new kinds of objects-built-from-Python-files > > that aren't modules at all. Do we need a new metaphor? Do we need one > > Magic But Really Well Spec'ed import hook that will satisfy everyone's > > needs? > > PEP 302 was intended to unify the import hook landscape, and make it > easier for hooks to work together. My impression is that it failed, > but I'm not sure why. My name is on the PEP, but I really only did > documentation, testing, and a little bit of design. Most of the good > stuff was Just's. > > I'd like to think that "tidying up" PEP 302 would solve the issues, > but I don't really have a good use case - my interest was with regard > to zip imports, and the sort of areas that eggs are addressing, but I > was looking forward to a situation where PEP 302 hooks were > established, and there would be no need to cater for custom > approaches. Sadly, I no longer see that day approaching :-( It's a really tough problem by now because the pile of import hacks just keeps growing. Probably we will need to lock the kinds of Tim Peters, Just van Rossum and Samuele Pedroni into some cathedral and wait until white smoke comes out of the chimney. > An immediate question springs to mind - what is the py magic doing, > and how does PEP 302 fail to cater for its needs? (Repeat this > question for any other package with nasty import magic). The py lib aims at compatibility with 2.2 upwards, so PEP 302 cannot really help much, even less so modifications to python-cvs at this point. Armin and me always had the feeling that PEP 302 could be used to cater for our needs, but never got around to thinking it through ... (for me, mainly the first issue kept me from having serious interest). FWIW, i think it is hampering Python development that the "standard library" only advances with major python releases. However, we begin to have the pieces in hand to head towards a more distributed approach of "batteries development". For me, the deployment/distribution/packaging/importing aspects seem like the single most deficient concepts with Python right now. (Everything else requires PyPy, anyway :-). cheers, holger From dripton at ripton.net Fri Apr 8 16:20:47 2005 From: dripton at ripton.net (David Ripton) Date: Fri, 8 Apr 2005 07:20:47 -0700 Subject: [py-dev] Does each test method get called with a new instance? In-Reply-To: <20050408084829.GA31081@vicky.ecs.soton.ac.uk> References: <20050405131933.GE23818@solar.trillke.net> <4252FF1C.10207@panix.com> <20050405213411.GM23818@solar.trillke.net> <20050407112940.GA16477@vicky.ecs.soton.ac.uk> <20050407211715.GG23818@solar.trillke.net> <1e199c56071ad7720ab53ce17a3feefb@panix.com> <20050407231821.GM23818@solar.trillke.net> <20050408024044.GA1576@vidar.dreamhost.com> <2mzmw97k85.fsf@starship.python.net> <20050408084829.GA31081@vicky.ecs.soton.ac.uk> Message-ID: <20050408142046.GA32623@vidar.dreamhost.com> On 2005.04.08 09:48:29 +0000, Armin Rigo wrote: > On Fri, Apr 08, 2005 at 09:20:42AM +0100, Michael Hudson wrote: > > > shouldn't be any grunt work either -- a unittest_to_pytest script would > > > be nice to have, even if it wasn't 100% perfect.) > > > > Err, we have one of these already, for PyPy. I don't know if it's > > generally useful, though. > > http://codespeak.net/svn/pypy/dist/pypy/tool/utestconvert.py Thanks for the pointer. > I think that apart from 5 lines for extra conversion cases specific to PyPy, > the tool is pretty much general and should be available somewhere with the py > lib. If it's not, 5 people will reinvent it. :-> -- David Ripton dripton at ripton.net From huymail at swiftdsl.com.au Fri Apr 8 18:42:59 2005 From: huymail at swiftdsl.com.au (Huy) Date: Sat, 09 Apr 2005 02:42:59 +1000 Subject: [py-dev] unitest.main() equivalent In-Reply-To: <20050407232856.GO23818@solar.trillke.net> References: <4255BB7C.9030300@swiftdsl.com.au> <20050407232856.GO23818@solar.trillke.net> Message-ID: <4256B493.9070804@swiftdsl.com.au> holger krekel wrote: > Hi Huy, > > On Fri, Apr 08, 2005 at 09:00 +1000, Huy wrote: > >>I was wondering if there is a unittest.main() equivalent in py.test > > > not really. > >>I'm basically trying to run my test directly in eclipse using pydev. > > > 'pydev' means py.test, i guess? Is it so that one runs pydev is the eclipse plugin for python development. It would be cool if py.test was some how integrated with this. pydev parses exception messages and auto links them to the source. > tests with eclipse by running the file directly? > Isn't there a way to customize eclipse by specifying that > "python path/to/py.test FILE_OR_DIRECTORY" > should be run for testing? Tried this but I want to run it through the pydev facilities, which work by running the test file itself. I can't get it to run pytest.bat (I'm on windows) conveniently with my test files/directory and still get the nice source linking. Thanks for your answers, Huy From hpk at trillke.net Fri Apr 8 19:48:07 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 8 Apr 2005 19:48:07 +0200 Subject: [py-dev] unitest.main() equivalent In-Reply-To: <4256B493.9070804@swiftdsl.com.au> References: <4255BB7C.9030300@swiftdsl.com.au> <20050407232856.GO23818@solar.trillke.net> <4256B493.9070804@swiftdsl.com.au> Message-ID: <20050408174807.GE23818@solar.trillke.net> Hi Huy, > holger krekel wrote: > >On Fri, Apr 08, 2005 at 09:00 +1000, Huy wrote: > >>I'm basically trying to run my test directly in eclipse using pydev. > > > >'pydev' means py.test, i guess? Is it so that one runs > > pydev is the eclipse plugin for python development. It would be cool if > py.test was some how integrated with this. pydev parses exception > messages and auto links them to the source. I see. > >tests with eclipse by running the file directly? > >Isn't there a way to customize eclipse by specifying that > >"python path/to/py.test FILE_OR_DIRECTORY" > >should be run for testing? > > Tried this but I want to run it through the pydev facilities, which work > by running the test file itself. I can't get it to run pytest.bat (I'm > on windows) conveniently with my test files/directory and still get the > nice source linking. Actually, I don't believe that py.test fully runs on Jython at the moment. Did you have _any_ success running it? (irrespective if source linking works or not). I would expect that at least assertion-expressions don't show nicely ... cheers, holger From pedronis at strakt.com Fri Apr 8 19:53:03 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Fri, 08 Apr 2005 19:53:03 +0200 Subject: [py-dev] unitest.main() equivalent In-Reply-To: <20050408174807.GE23818@solar.trillke.net> References: <4255BB7C.9030300@swiftdsl.com.au> <20050407232856.GO23818@solar.trillke.net> <4256B493.9070804@swiftdsl.com.au> <20050408174807.GE23818@solar.trillke.net> Message-ID: <4256C4FF.5000207@strakt.com> holger krekel wrote: >Actually, I don't believe that py.test fully runs on Jython >at the moment. Did you have _any_ success running it? >(irrespective if source linking works or not). I would expect >that at least assertion-expressions don't show nicely ... > > > pydev by default run things through an external CPython, nothing to do with Jython. One can change that, I don't know whether configuring different runners at the same time is easy. From hpk at trillke.net Sat Apr 9 10:42:58 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 9 Apr 2005 10:42:58 +0200 Subject: [py-dev] unitest.main() equivalent In-Reply-To: <4256C4FF.5000207@strakt.com> References: <4255BB7C.9030300@swiftdsl.com.au> <20050407232856.GO23818@solar.trillke.net> <4256B493.9070804@swiftdsl.com.au> <20050408174807.GE23818@solar.trillke.net> <4256C4FF.5000207@strakt.com> Message-ID: <20050409084258.GF23818@solar.trillke.net> On Fri, Apr 08, 2005 at 19:53 +0200, Samuele Pedroni wrote: > holger krekel wrote: > > >Actually, I don't believe that py.test fully runs on Jython > >at the moment. Did you have _any_ success running it? > >(irrespective if source linking works or not). I would expect > >that at least assertion-expressions don't show nicely ... > > > pydev by default run things through an external CPython, nothing to do > with Jython. ah, thanks for the info. That should make it easy enough to integrate py.test's mechnisms into pydev at some point. > One can change that, I don't know whether configuring different runners > at the same time is easy. I don't know either but will have a look into pydev at some point but that will take some time. holger P.S.: "pydev" is a pretty confusing name here on py-dev :-) From hpk at trillke.net Sun Apr 10 12:57:24 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 10 Apr 2005 12:57:24 +0200 Subject: [py-dev] Re: reponsiveness of tkinter frontend In-Reply-To: <4258FFD7.5040005@balster.info> References: <20050409193540.GL23818@solar.trillke.net> <4258FFD7.5040005@balster.info> Message-ID: <20050410105724.GM23818@solar.trillke.net> Hi Jan, (CCed to py-dev because other might be interested yadda yadda) On Sun, Apr 10, 2005 at 12:28 +0200, Jan Balster wrote: > holger krekel wrote: > > Hi Jan, > > > > i was wondering if you know why tkinter does not > > remain responsive throughout the testing process. > > I have been using it a couple of times but felt > > that the UI often blocked. > > The tree is updated for every call to guidriver.start and guidriver.end. > If you can see many tree items, it's very slow. That's a problem with > the tktree package I use. It's not possible for to just update part of the tree? > I can write quickly a simple frontend with a list for failed and skipped > tests if you need a more responsive gui. I am not sure exactly here. But I think a simple list of failed tests with a fully responsive GUI (so i can inspect failures while testing continues seamlessly) would be a good idea. Btw, ExceptionFailures are subclasses of Failed and should be considered plain failures. The distinction only exists for reporting purposes (exception failures show differently than "normal" assertion failures or runtime Exceptions). So if i may wish for something: i would offer checkboxes for failed [x] skipped [ ] passed [ ] with only "failed" tests being shown by default. A status line (as well as the title of the windows, probably) shows the full test path being run at the moment. Note that i changed the default output for "py.test" to now show full module names. This became possible because the import system / logic got sanitized a lot and basically every test now always has a full package path, no matter if you run it directly or from some higher up directory. This uniformity is important for customization purposes. Also soon coming (i hope :-) is the feature i mention here: http://codespeak.net/py/current/doc/test.html#selecting-tests-by-queries-full-text-search The according tkinter interface could be a google style textfield where you can change the set of "selected tests" by dynamically specifying queries. The list of tests could even be updated after every key stroke ("incremental search"). I am just mentioning this because it might play a role in your tkinter internal design decisions. > > Another issue was, that when i clicked on a failure > > to investigate a problem that the ongoing testing > > process kept resetting the bottom window's focus > > to the first line. > > The tree widget remembers which item was selected before the tests are > executed and trys to reselect the item when it's recreated (test is > executed). So that shouldn't happen. I can't reproduce your result, can > you describe exactly what you did? Next time i have the problem, i try to describe it in more detail. thanks & cheers, holger From hpk at trillke.net Sun Apr 10 21:15:04 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 10 Apr 2005 21:15:04 +0200 Subject: [py-dev] Re: reponsiveness of tkinter frontend In-Reply-To: <42591467.9020906@balster.info> References: <20050409193540.GL23818@solar.trillke.net> <4258FFD7.5040005@balster.info> <20050410105724.GM23818@solar.trillke.net> <42591467.9020906@balster.info> Message-ID: <20050410191504.GW23818@solar.trillke.net> Hi Jan, On Sun, Apr 10, 2005 at 13:56 +0200, Jan Balster wrote: > holger krekel wrote: > > On Sun, Apr 10, 2005 at 12:28 +0200, Jan Balster wrote: > >>The tree is updated for every call to guidriver.start and guidriver.end. > >>If you can see many tree items, it's very slow. That's a problem with > >>the tktree package I use. > > > > It's not possible for to just update part of the tree? > > > The tree widget can't change a simple node into a expandable node (node > with child nodes). > The tree uses a callback to build nodes on demand. If a collector > starts, the coressponding node has no childnodes. If the next test > starts, the node has to be changed into a expandable node. To do this > the parent node will be collapsed and expanded again. The callback will > build the 'collector'node as an expandable node, because it has childtests. I think i understand. It's unfortunate, then, that the Tktree widget can't efficiently handle growing trees. > >>I can write quickly a simple frontend with a list for failed and skipped > >>tests if you need a more responsive gui. That would be useful. If possible, i'd put this into a separate tab. Keeping a full tree view of all tests (with failing/passing/skipped just by color) seems worthwhile. But the "failing test list" with the checkboxes should be the default. Note that the search-query-interface, i mentioned, is not there yet. cheers, holger From jan at balster.info Sun Apr 10 22:48:34 2005 From: jan at balster.info (Jan Balster) Date: Sun, 10 Apr 2005 22:48:34 +0200 Subject: [py-dev] Re: reponsiveness of tkinter frontend In-Reply-To: <20050410191504.GW23818@solar.trillke.net> References: <20050409193540.GL23818@solar.trillke.net> <4258FFD7.5040005@balster.info> <20050410105724.GM23818@solar.trillke.net> <42591467.9020906@balster.info> <20050410191504.GW23818@solar.trillke.net> Message-ID: <42599122.10705@balster.info> holger krekel wrote: I just saw the new --tkinter option. HipHipHurray! Can I use --session=guisession on the remote side? What about options for a sessionclass? >>>>I can write quickly a simple frontend with a list for failed and skipped >>>>tests if you need a more responsive gui. > > > That would be useful. If possible, i'd put this into a separate tab. > Keeping a full tree view of all tests (with failing/passing/skipped > just by color) seems worthwhile. But the "failing test list" with > the checkboxes should be the default. I'm working on this. Jan From hpk at trillke.net Sun Apr 10 23:44:18 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 10 Apr 2005 23:44:18 +0200 Subject: [py-dev] Re: reponsiveness of tkinter frontend In-Reply-To: <42599122.10705@balster.info> References: <20050409193540.GL23818@solar.trillke.net> <4258FFD7.5040005@balster.info> <20050410105724.GM23818@solar.trillke.net> <42591467.9020906@balster.info> <20050410191504.GW23818@solar.trillke.net> <42599122.10705@balster.info> Message-ID: <20050410214418.GX23818@solar.trillke.net> Hi Jan, On Sun, Apr 10, 2005 at 22:48 +0200, Jan Balster wrote: > holger krekel wrote: > I just saw the new --tkinter option. HipHipHurray! :-) i am still refactoring the exact details a bit ... > Can I use --session=guisession on the remote side? not quite. --session is just a user-level option. How we internally organize our sessions is another issue. I intend to now get --looponfailing and --exec back to run and take care that i have a clean approach "splitting" sessions into a local and remote side. This should be a good guide to the tkinter handling (which i guess could be simplified a lot, but may require small extensions to py.execnet). > What about options for a sessionclass? if you look into cmdline.py you'll find at the beginning: def main(): config, args = py.test.Config.parse(py.std.sys.argv[1:]) this config object is an instance of py/test/config.py's Config class and has an "option" attribute, holding all options (including any custom ones by applications in which context we run). When session.main() is invoked, it also has such a 'config' attribute. I want all collectors and items to grow it too. This evening's refactorings got actually triggered by wanting to fully implement the already existing --traceconfig option which requires the collectors to have access to configs. Having tried this earlier today brought me the N'th time to the config mess, which i hope is now in an acceptable state. > >>>>I can write quickly a simple frontend with a list for failed and skipped > >>>>tests if you need a more responsive gui. > > > > That would be useful. If possible, i'd put this into a separate tab. > > Keeping a full tree view of all tests (with failing/passing/skipped > > just by color) seems worthwhile. But the "failing test list" with > > the checkboxes should be the default. > > I'm working on this. great. i am still trying to finalize session-handling details but will not touch tkinter much this night. Btw, i am on #pylib (a freshly opened irc.freenode.net IRC channel :-) in case you want to talk more interactively ... cheers, holger From briandorsey at gmail.com Tue Apr 12 17:57:08 2005 From: briandorsey at gmail.com (Brian Dorsey) Date: Tue, 12 Apr 2005 08:57:08 -0700 Subject: [py-dev] Re: [py-svn] r10550 - py/dist/py/path/local In-Reply-To: <20050412142409.A713A27B48@code1.codespeak.net> References: <20050412142409.A713A27B48@code1.codespeak.net> Message-ID: <66e877b70504120857605b5c82@mail.gmail.com> Armin, This change breaks py.test on windows for me, since os.path.samefile doesn't seem to exist in windows. Take care, -Brian On Apr 12, 2005 7:24 AM, arigo at codespeak.net wrote: > Author: arigo > Date: Tue Apr 12 16:24:09 2005 > New Revision: 10550 > > Modified: > py/dist/py/path/local/local.py > Log: > Oups. Crashed if the co_filename stored in .pyc files doesn't exist at all. > > Modified: py/dist/py/path/local/local.py > ============================================================================== > --- py/dist/py/path/local/local.py (original) > +++ py/dist/py/path/local/local.py Tue Apr 12 16:24:09 2005 > @@ -347,8 +347,13 @@ > co = py.std.marshal.load(f) > path1 = co.co_filename > path2 = str(self) > - if path1 == path2 or os.path.samefile(path1, path2): > + if path1 == path2: > return co > + try: > + if os.path.samefile(path1, path2): > + return co > + except OSError: # probably path1 not found > + pass > finally: > f.close() > except py.error.Error: > _______________________________________________ > py-svn mailing list > py-svn at codespeak.net > http://codespeak.net/mailman/listinfo/py-svn > From briandorsey at gmail.com Tue Apr 12 18:34:45 2005 From: briandorsey at gmail.com (Brian Dorsey) Date: Tue, 12 Apr 2005 09:34:45 -0700 Subject: [py-dev] Re: [py-svn] r10550 - py/dist/py/path/local In-Reply-To: <66e877b70504120857605b5c82@mail.gmail.com> References: <20050412142409.A713A27B48@code1.codespeak.net> <66e877b70504120857605b5c82@mail.gmail.com> Message-ID: <66e877b7050412093436d9b82@mail.gmail.com> And, actually I didn't read this correctly, it looks like samefile was already there, I just hadn't run into the error until now. I'm not sure exactly how this code is used... would it be safe to skip this check on windows, or should I dig around to find some sort of equivalent version for windows? Take care, -Brian On Apr 12, 2005 8:57 AM, Brian Dorsey wrote: > Armin, > > This change breaks py.test on windows for me, since os.path.samefile > doesn't seem to exist in windows. > > Take care, > -Brian > > On Apr 12, 2005 7:24 AM, arigo at codespeak.net wrote: > > Author: arigo > > Date: Tue Apr 12 16:24:09 2005 > > New Revision: 10550 > > > > Modified: > > py/dist/py/path/local/local.py > > Log: > > Oups. Crashed if the co_filename stored in .pyc files doesn't exist at all. > > > > Modified: py/dist/py/path/local/local.py > > ============================================================================== > > --- py/dist/py/path/local/local.py (original) > > +++ py/dist/py/path/local/local.py Tue Apr 12 16:24:09 2005 > > @@ -347,8 +347,13 @@ > > co = py.std.marshal.load(f) > > path1 = co.co_filename > > path2 = str(self) > > - if path1 == path2 or os.path.samefile(path1, path2): > > + if path1 == path2: > > return co > > + try: > > + if os.path.samefile(path1, path2): > > + return co > > + except OSError: # probably path1 not found > > + pass > > finally: > > f.close() > > except py.error.Error: > > _______________________________________________ > > py-svn mailing list > > py-svn at codespeak.net > > http://codespeak.net/mailman/listinfo/py-svn > > > From hpk at trillke.net Tue Apr 12 21:33:08 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 12 Apr 2005 21:33:08 +0200 Subject: [py-dev] Re: [py-svn] r10550 - py/dist/py/path/local In-Reply-To: <66e877b7050412093436d9b82@mail.gmail.com> References: <20050412142409.A713A27B48@code1.codespeak.net> <66e877b70504120857605b5c82@mail.gmail.com> <66e877b7050412093436d9b82@mail.gmail.com> Message-ID: <20050412193308.GL23818@solar.trillke.net> Hi Brian, On Tue, Apr 12, 2005 at 09:34 -0700, Brian Dorsey wrote: > And, actually I didn't read this correctly, it looks like samefile was > already there, I just hadn't run into the error until now. I'm not > sure exactly how this code is used... would it be safe to skip this > check on windows, or should I dig around to find some sort of > equivalent version for windows? Armin seems to have fixed it already (as you may have noticed on py-svn, but just because someone else mailed me on the issue already). cheers & greetings to seattle! holger From briandorsey at gmail.com Wed Apr 13 05:22:05 2005 From: briandorsey at gmail.com (Brian Dorsey) Date: Tue, 12 Apr 2005 20:22:05 -0700 Subject: [py-dev] Re: [py-svn] r10550 - py/dist/py/path/local In-Reply-To: <20050412193308.GL23818@solar.trillke.net> References: <20050412142409.A713A27B48@code1.codespeak.net> <66e877b70504120857605b5c82@mail.gmail.com> <66e877b7050412093436d9b82@mail.gmail.com> <20050412193308.GL23818@solar.trillke.net> Message-ID: <66e877b705041220227cefe4f0@mail.gmail.com> On 4/12/05, holger krekel wrote: > Armin seems to have fixed it already (as you may have noticed on > py-svn, but just because someone else mailed me on the issue already). Thanks Armin & Holger! That seems to work for me. Take care, -Brian From arigo at tunes.org Wed Apr 13 18:33:30 2005 From: arigo at tunes.org (Armin Rigo) Date: Wed, 13 Apr 2005 17:33:30 +0100 Subject: [py-dev] Greenlets fixed Message-ID: <20050413163330.GA25466@vicky.ecs.soton.ac.uk> Hi, 'py.magic.greenlet' wasn't working properly on MacOSX. This notice is to tell you that I found and fixed the problem. It wasn't specific to OSX, but to various details of the compiler and the architecture, so it showed up on OSX and by chance not on Linux/PC. If you had problems on other platforms (typically getting RuntimeError: maximum recursion depth exceeded), it would have been fixed too. Armin From jan at balster.info Thu Apr 14 16:18:15 2005 From: jan at balster.info (Jan Balster) Date: Thu, 14 Apr 2005 16:18:15 +0200 Subject: [py-dev] new Channel callback Message-ID: <425E7BA7.9050806@balster.info> Hi Holger, if I use the new channel.setcallback I still have to call gateway.exit() at the end. So I need a thread to wait for the remote side to finish. Is there a way to avoid this? Calling gateway.exit() form the callback would be fine for me, but I'm afraid that won't work. Jan From hpk at trillke.net Thu Apr 14 17:16:35 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 14 Apr 2005 17:16:35 +0200 Subject: [py-dev] new Channel callback In-Reply-To: <425E7BA7.9050806@balster.info> References: <425E7BA7.9050806@balster.info> Message-ID: <20050414151635.GY23818@solar.trillke.net> Hi Jan, On Thu, Apr 14, 2005 at 16:18 +0200, Jan Balster wrote: > if I use the new channel.setcallback I still have to call gateway.exit() > at the end. So I need a thread to wait for the remote side to finish. > > Is there a way to avoid this? Calling gateway.exit() form the callback > would be fine for me, but I'm afraid that won't work. Maybe that should work, although the gateway/channel shutdown-process is rather involved so it might actually not work. It's worth just testing it out, i'd say. cheers, holger From jan at balster.info Thu Apr 14 17:41:05 2005 From: jan at balster.info (Jan Balster) Date: Thu, 14 Apr 2005 17:41:05 +0200 Subject: [py-dev] new Channel callback In-Reply-To: <20050414151635.GY23818@solar.trillke.net> References: <425E7BA7.9050806@balster.info> <20050414151635.GY23818@solar.trillke.net> Message-ID: <425E8F11.7020606@balster.info> holger krekel wrote: > Hi Jan, > > On Thu, Apr 14, 2005 at 16:18 +0200, Jan Balster wrote: > >>if I use the new channel.setcallback I still have to call gateway.exit() >>at the end. So I need a thread to wait for the remote side to finish. >> >>Is there a way to avoid this? Calling gateway.exit() form the callback >>would be fine for me, but I'm afraid that won't work. > > > Maybe that should work, although the gateway/channel shutdown-process > is rather involved so it might actually not work. It's worth > just testing it out, i'd say. Hurray! It works. But it seems to me, using the callback is slower than receiving all items in a loop (13 secs vs. 15 secs). Jan From hpk at trillke.net Thu Apr 14 17:49:35 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 14 Apr 2005 17:49:35 +0200 Subject: [py-dev] new Channel callback In-Reply-To: <425E8F11.7020606@balster.info> References: <425E7BA7.9050806@balster.info> <20050414151635.GY23818@solar.trillke.net> <425E8F11.7020606@balster.info> Message-ID: <20050414154935.GZ23818@solar.trillke.net> On Thu, Apr 14, 2005 at 17:41 +0200, Jan Balster wrote: > holger krekel wrote: > > On Thu, Apr 14, 2005 at 16:18 +0200, Jan Balster wrote: > >>if I use the new channel.setcallback I still have to call gateway.exit() > >>at the end. So I need a thread to wait for the remote side to finish. > >> > >>Is there a way to avoid this? Calling gateway.exit() form the callback > >>would be fine for me, but I'm afraid that won't work. > > > > > > Maybe that should work, although the gateway/channel shutdown-process > > is rather involved so it might actually not work. It's worth > > just testing it out, i'd say. > > Hurray! It works. But it seems to me, using the callback is slower than > receiving all items in a loop (13 secs vs. 15 secs). That would be very odd. Do you repeatedly get this? Btw, i'd recommend to add a test for exiting from a callback because i am currently revising the threading mechanism :-) cheers, holger From jan at balster.info Thu Apr 14 19:17:37 2005 From: jan at balster.info (Jan Balster) Date: Thu, 14 Apr 2005 19:17:37 +0200 Subject: [py-dev] new Channel callback In-Reply-To: <20050414154935.GZ23818@solar.trillke.net> References: <425E7BA7.9050806@balster.info> <20050414151635.GY23818@solar.trillke.net> <425E8F11.7020606@balster.info> <20050414154935.GZ23818@solar.trillke.net> Message-ID: <425EA5B1.4080308@balster.info> holger krekel wrote: > On Thu, Apr 14, 2005 at 17:41 +0200, Jan Balster wrote: > >>holger krekel wrote: >> >>>On Thu, Apr 14, 2005 at 16:18 +0200, Jan Balster wrote: >>> >>>>if I use the new channel.setcallback I still have to call gateway.exit() >>>>at the end. So I need a thread to wait for the remote side to finish. >>>> >>>>Is there a way to avoid this? Calling gateway.exit() form the callback >>>>would be fine for me, but I'm afraid that won't work. >>> >>> >>>Maybe that should work, although the gateway/channel shutdown-process >>>is rather involved so it might actually not work. It's worth >>>just testing it out, i'd say. >> >>Hurray! It works. But it seems to me, using the callback is slower than >>receiving all items in a loop (13 secs vs. 15 secs). > > > That would be very odd. Do you repeatedly get this? > Btw, i'd recommend to add a test for exiting from a callback > because i am currently revising the threading mechanism :-) Ok. I did some benchmarking. Callback is faster than Loop, except for small numbers of received items. Jan benchmark output: Receiving 10000 elements use_callback: 3.702 use_loop: 4.522 ******************** The winner is Callback Receiving 2000 elements use_callback: 0.919 use_loop: 1.175 ******************** The winner is Callback Receiving 1000 elements use_callback: 0.573 use_loop: 0.588 ******************** The winner is Callback Receiving 100 elements use_callback: 0.270 use_loop: 0.250 ******************** The winner is Loop Receiving 50 elements use_callback: 0.274 use_loop: 0.235 ******************** The winner is Loop Receiving 10 elements use_callback: 0.230 use_loop: 0.219 ******************** The winner is Loop close failed: [Errno 9] Bad file descriptor program: import py remote_code = ''' import sha number = channel.receive() for i in range(number): s = sha.new(str(i)).hexdigest() * 10 channel.send(s) ''' def use_callback(number): gw = py.execnet.PopenGateway(py.std.sys.executable) channel = gw.newchannel() l = [] channel.setcallback(l.append) channel.remote_exec(remote_code) channel.send(number) channel.waitclose(100.0) channel.close() channel.gateway.exit() def use_loop(number): gw = py.execnet.PopenGateway(py.std.sys.executable) channel = gw.newchannel() l = [] channel.remote_exec(remote_code) channel.send(number) try: while 1: l.append(channel.receive()) except EOFError: pass channel.waitclose(100.0) channel.close() channel.gateway.exit() def test(number): print print 'Receiving', number, 'elements' t1 = py.std.time.time() use_callback(number) t1 = py.std.time.time() - t1 print 'use_callback: \t%0.3f' % t1 t2 = py.std.time.time() use_loop(number) t2 = py.std.time.time() - t2 print 'use_loop: \t%0.3f' % t2 print '*'* 20, 'The winner is', if t1 < t2: print 'Callback' else: print 'Loop' if __name__ == '__main__': for i in (10000, 2000, 1000, 100, 50, 10): test(i) From hpk at trillke.net Thu Apr 14 20:05:43 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 14 Apr 2005 20:05:43 +0200 Subject: [py-dev] new Channel callback In-Reply-To: <425EA5B1.4080308@balster.info> References: <425E7BA7.9050806@balster.info> <20050414151635.GY23818@solar.trillke.net> <425E8F11.7020606@balster.info> <20050414154935.GZ23818@solar.trillke.net> <425EA5B1.4080308@balster.info> Message-ID: <20050414180543.GB23818@solar.trillke.net> Hi Jan, On Thu, Apr 14, 2005 at 19:17 +0200, Jan Balster wrote: > holger krekel wrote: > > On Thu, Apr 14, 2005 at 17:41 +0200, Jan Balster wrote: > >>Hurray! It works. But it seems to me, using the callback is slower than > >>receiving all items in a loop (13 secs vs. 15 secs). > > > > That would be very odd. Do you repeatedly get this? > > Btw, i'd recommend to add a test for exiting from a callback > > because i am currently revising the threading mechanism :-) > Ok. > > I did some benchmarking. Callback is faster than Loop, except for small > numbers of received items. Hum, i note that you are also measuring creation/teardown time. I've attached a modified benchmark script which avoids this, reusing the same gateway all along. There, for small number of elements, the behaviour is not completly consistent anymore (callback sometimes wins). However, I don't currently care anyhow as callbacks are not there for speed reasons but for convenience. They might even become slower when setting a callback would run that callback in a separate thread which might make sense as a default behaviour (otherwise you could block the receiver thread where callbacks are currently run). In real life applications (like the tkinter frontend) the callback gains you that you don't have to start a thread just to push an item from one queue to another. cheers, holger import py remote_code = ''' import sha number = channel.receive() for i in range(number): s = sha.new(str(i)).hexdigest() * 10 channel.send(s) ''' def use_callback(gw, number): channel = gw.newchannel() l = [] channel.setcallback(l.append) channel.remote_exec(remote_code) channel.send(number) channel.waitclose(100.0) assert len(l) == number def use_loop(gw, number): channel = gw.newchannel() l = [] channel.remote_exec(remote_code) channel.send(number) l = list(channel) channel.waitclose(100.0) assert len(l) == number def test(gw, number): print print 'Receiving', number, 'elements' t1 = py.std.time.time() use_callback(gw, number) t1 = py.std.time.time() - t1 print 'use_callback: \t%0.3f' % t1 t2 = py.std.time.time() use_loop(gw, number) t2 = py.std.time.time() - t2 print 'use_loop: \t%0.3f' % t2 print '*'* 20, 'The winner is', if t1 < t2: print 'Callback' else: print 'Loop' if __name__ == '__main__': gw = py.execnet.PopenGateway(py.std.sys.executable) print "created gateway" for i in (10000, 2000, 1000, 100, 50, 10): test(gw, i) gw.exit() From wam at cisco.com Thu Apr 14 23:42:07 2005 From: wam at cisco.com (William McVey) Date: Thu, 14 Apr 2005 16:42:07 -0500 Subject: [py-dev] py.xml.html Message-ID: <1113514927.3054.17.camel@localhost> I've recently started using the py.xml.html package and ran across some additional HTML elements that would be nice to see get added to the defined __tagspec__ of the html class. Namely: "colgroup,col,map,area,blockquote,dl,dt,dd" "strong,script,style" Also, it would be nice if there were a flag that could turn off the XML short cut of . In particular, when generating HTML for older browsers, it is much better to generate elements using the syntax of . I've hacked this feature into xmlobj/visit.py in a very non-elegant fashion (always favoring the HTML-friendly approach, which works for me, but probably isn't appropriate for all cases). Let me know if other people think this is a good idea, and I can try to polish this up based off of an instantiation flag or something. -- William From hpk at trillke.net Fri Apr 15 00:11:11 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 15 Apr 2005 00:11:11 +0200 Subject: [py-dev] py.xml.html In-Reply-To: <1113514927.3054.17.camel@localhost> References: <1113514927.3054.17.camel@localhost> Message-ID: <20050414221111.GD23818@solar.trillke.net> Hi William, On Thu, Apr 14, 2005 at 16:42 -0500, William McVey wrote: > I've recently started using the py.xml.html package and ran across some > additional HTML elements that would be nice to see get added to the > defined __tagspec__ of the html class. Namely: > "colgroup,col,map,area,blockquote,dl,dt,dd" > "strong,script,style" thanks, added (apart from script and style, which already existed in my tree). > Also, it would be nice if there were a flag that could turn off the XML > short cut of . In particular, when generating HTML for > older browsers, it is much better to generate elements using the syntax > of . I've hacked this feature into > xmlobj/visit.py in a very non-elegant fashion (always favoring the > HTML-friendly approach, which works for me, but probably isn't > appropriate for all cases). Let me know if other people think this is a > good idea, and I can try to polish this up based off of an instantiation > flag or something. This sounds familiar. I just added a flag to the Core visitor (shortempty) and made html turn that off by default. So the tags should now serialize as you expect. A question: are you also using the CSS features? And please feel free to make suggestions and contributions. cheers, holger From wam at cisco.com Fri Apr 15 01:15:49 2005 From: wam at cisco.com (William McVey) Date: Thu, 14 Apr 2005 18:15:49 -0500 Subject: [py-dev] py.xml.html In-Reply-To: <20050414221111.GD23818@solar.trillke.net> References: <1113514927.3054.17.camel@localhost> <20050414221111.GD23818@solar.trillke.net> Message-ID: <1113520549.3054.36.camel@localhost> On Thu, 2005-04-14 at 17:11, holger krekel wrote: > thanks, added (apart from script and style, which already existed in my tree). > I just added a flag to the Core visitor... Wow, talk about responsive to suggestions. Thanks! > A question: are you also using the CSS features? Nope... We have a set of external stylesheets which have already been created, so I'm just using py.xml.html to create my elements with the expected class and id attributes. > And please feel free to make suggestions and contributions. Will do... thanks. At this point, I'm only using a very small fraction of the py library. If/when I start integrating its other features, I may have more to feed back to the community. -- William P.S. I know one of the goals of the py library is to go beyond the 'cut a release and distribute it' model, but it would be nice if there were a distutil install script. From hpk at trillke.net Fri Apr 15 16:11:13 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 15 Apr 2005 16:11:13 +0200 Subject: [py-dev] Re: [py-svn] r10665 - in py/branch/py-collect: . test test/terminal test/testing In-Reply-To: <425FBE76.5020301@balster.info> References: <20050415114330.1871527B7A@code1.codespeak.net> <425FBE76.5020301@balster.info> Message-ID: <20050415141113.GL23818@solar.trillke.net> Hi Jan, (reposted to py-dev), On Fri, Apr 15, 2005 at 15:15 +0200, Jan Balster wrote: > hpk at codespeak.net wrote: > i think you missed a return staement in terminal.py: > > + def main(self, args): > > + if self.config.option._remote: > > + from py.__impl__.test.terminal import remote > > + remote.main(self.config, self._file, self.config._origargs) > > + else: > > + super(TerminalSession, self).main(args) good catch, i didn't test --looponfailing so far which needs the return value. I actually want to add a test for looonfailing and will fix it then (it's good if tests fail first :-). btw, note that i didn't really get to do the clean implementation of "remote driving". It's still the old stdout/stderr redirection thingie, basically (but at least remote_exec() now directly allows to catch stdout/stderr per invocation, even if you have concurrent remote_exec() invocations :-). The thing is that i am pretty sure that execnet needs to support send/receive of (at least) simple objects. But maybe we should first try have a "pickle"-layer on top of channels because we want to assume anyway that the classes and the import structure are equivalent on the local and the remote side. With such an assumption we can probably send instances over the channel via pickle. cheers, holger From pobrien at orbtech.com Sat Apr 16 17:55:32 2005 From: pobrien at orbtech.com (Patrick K. O'Brien) Date: Sat, 16 Apr 2005 10:55:32 -0500 Subject: [py-dev] Re: [py-svn] r10665 - in py/branch/py-collect: . test test/terminal test/testing In-Reply-To: <20050415141113.GL23818@solar.trillke.net> References: <20050415114330.1871527B7A@code1.codespeak.net> <425FBE76.5020301@balster.info> <20050415141113.GL23818@solar.trillke.net> Message-ID: <42613574.1040404@orbtech.com> holger krekel wrote: > > The thing is that i am pretty sure that execnet needs to support > send/receive of (at least) simple objects. But maybe we should first > try have a "pickle"-layer on top of channels because we want to > assume anyway that the classes and the import structure are > equivalent on the local and the remote side. With such an > assumption we can probably send instances over the channel > via pickle. Sometimes you don't want to have to have equivalent import structures on the remote side, or you don't want to be limited by Python's import mechanisms, or you don't even want the class definitions to exist as physical files on the remote side. We've run into all these situations with Schevo. So I'd love to collaborate on solutions that would work for everyone, assuming there is an interest in such things. We have some code in Schevo that might be worth looking at. Feel free to take any ideas that look worthwhile. Here are some good starting points for code that we have that deals with this issue: http://svn.orbtech.com/cgi-bin/schevo.cgi/file/trunk/schevo/module.py http://svn.orbtech.com/cgi-bin/schevo.cgi/browser/trunk/schevo/schema/ -- Patrick K. O'Brien Orbtech http://www.orbtech.com Schevo http://www.schevo.org Pypersyst http://www.pypersyst.org From hpk at trillke.net Sat Apr 16 21:57:51 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 16 Apr 2005 21:57:51 +0200 Subject: [py-dev] branch is being merged back Message-ID: <20050416195750.GS23818@solar.trillke.net> Hi all, we are currently merging back the py-collect branch [1]. Although this is a huge diff we don't expect big problems unless you are using "conftest.py" customizations. There are a couple of new command line options, most notably Jan Balster's "--tkinter" frontend. It's at an early experimental stage but already starts to be usable. Note that the command line options changed slightly, because there is a new rule: py.test will consume all lowercase single-letter options like "-h", "-v", "-x" and so on while applications should add only consume capital cased letters. One major change that took place in the branch is that we are refraining from import magic for importing test modules. This deprives us of the ability to run tests directly from a remote repository, but allows to get rid of some nasty interactions with e.g. nevow's import hacks. have fun and report any problems, holger [1] Jan, if you have a locally modified pycollect branch you will have to switch to the dist directory because the branch/py-collect is no more and you can't commit to it ... From dripton at ripton.net Mon Apr 18 15:52:36 2005 From: dripton at ripton.net (David Ripton) Date: Mon, 18 Apr 2005 06:52:36 -0700 Subject: [py-dev] branch is being merged back In-Reply-To: <20050416195750.GS23818@solar.trillke.net> References: <20050416195750.GS23818@solar.trillke.net> Message-ID: <20050418135236.GA21910@vidar.dreamhost.com> On 2005.04.16 21:57:51 +0000, holger krekel wrote: > Note that the command line options changed slightly, because > there is a new rule: py.test will consume all lowercase > single-letter options like "-h", "-v", "-x" and so on while > applications should add only consume capital cased letters. Haven't looked at the new code yet, so maybe I'm interpreting your statement incorrectly, but this seems wrong. I currently don't rely on the value of sys.argv in unit tests. If I need to test a script that uses argv, I split out a main function from the top-level "if __name__ == '__main__'" block and make the latter as small and simple as possible (usually just "main()") so I don't have to test it. Then I do: def main(argv=sys.argv) or sometimes even: def main(argv=None): if argv is None: argv = sys.argv So that I have the ability to plug in different argv values for testing, without actually hacking the "real" sys.argv This is occasionally inconvenient. For example, sometimes a script requires a password, and you don't want to hardcode it, and you don't want to put it in a file, and you don't want your test to be overly interactive, and putting it on the command line feels like the least insecure of the simple options. (I know, someone could use ps to sniff it, so this is hardly super-secure.) Now, it would be nice to assume that py.test will leave {some of} sys.argv alone, so that I have the option of using it. But a rule based on capitalization seems like the wrong way to do this. I want to pick option names for my programs based on what's user-friendly, not what my test framework allows. Maybe a better way to do this would be some kind of --the-rest-go-to-the-program-under-test option (with a shorter name), that py.test could use to split sys.argv. py.test would consume anything on the command line before the marker, and pass everything after it to the application as sys.argv -- David Ripton dripton at ripton.net From hpk at trillke.net Mon Apr 18 22:00:36 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 18 Apr 2005 22:00:36 +0200 Subject: [py-dev] branch is being merged back In-Reply-To: <20050418135236.GA21910@vidar.dreamhost.com> References: <20050416195750.GS23818@solar.trillke.net> <20050418135236.GA21910@vidar.dreamhost.com> Message-ID: <20050418200036.GX23818@solar.trillke.net> Hi David, On Mon, Apr 18, 2005 at 06:52 -0700, David Ripton wrote: > On 2005.04.16 21:57:51 +0000, holger krekel wrote: > > Note that the command line options changed slightly, because > > there is a new rule: py.test will consume all lowercase > > single-letter options like "-h", "-v", "-x" and so on while > > applications should add only consume capital cased letters. > > Haven't looked at the new code yet, so maybe I'm interpreting your > statement incorrectly, but this seems wrong. Sorry, i wasn't very precise in my above statement. py.test has always been using command line options. But, for example with PyPy, we have a couple of extra options to the testing process (which you can provide through 'conftest.py' files in your package). Now the idea is that such test extensions (what i called "applications" above) should use upper-case letters while py.test restrains itself to only use lower-case ones. This way py.test has enough future possible options :-) So with respect to what a test module finds in sys.argv nothing has changed. In fact, if a test module inspects sys.argv it will find the complete list of options (provided to py.test). > I currently don't rely on the value of sys.argv in unit tests. If I need > to test a script that uses argv, I split out a main function from the > top-level "if __name__ == '__main__'" block and make the latter as small > and simple as possible (usually just "main()") so I don't have to test it. makes sense. > ... > This is occasionally inconvenient. For example, sometimes a script > requires a password, and you don't want to hardcode it, and you don't > want to put it in a file, and you don't want your test to be overly > interactive, and putting it on the command line feels like the least > insecure of the simple options. (I know, someone could use ps to sniff > it, so this is hardly super-secure.) > > Now, it would be nice to assume that py.test will leave {some of} > sys.argv alone, so that I have the option of using it. Yes, although it is not completely obvious how to exactly do it (also with respect to the underlying optparse-usage). > But a rule based on capitalization seems like the wrong way to do this. > I want to pick option names for my programs based on what's > user-friendly, not what my test framework allows. Sure but py.test is usually not running your application, is it? :-) Basically i would suggest that you add a special testing command line option that holds your password information. E.g. in a conftest.py in your package you could add: option = py.test.addptions("my app testing options", Option('-P', '--password', action="store", default=None, type="string", dest="password", help="password used for XXX connections."), And then in your test_whatever.py: from mypackage.conftest import option .... def test_something(): myscript.main(...use option.password... ) If you then are in your package and issue 'py.test -h' it will have your option. cheers, holger From ianb at colorstudy.com Fri Apr 22 05:38:03 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 21 Apr 2005 22:38:03 -0500 Subject: [py-dev] Import errors Message-ID: <4268719B.2000103@colorstudy.com> Well, I'm doing a big renaming, and the errors from bad imports are a bit confusing. It seems whenever there's an import error, the module with the error still gets loaded, but is empty. This is a bit confusing to debug. Well, this isn't true in the test_* module, but in the modules they import. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From hpk at trillke.net Fri Apr 22 10:15:50 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 22 Apr 2005 10:15:50 +0200 Subject: [py-dev] Import errors In-Reply-To: <4268719B.2000103@colorstudy.com> References: <4268719B.2000103@colorstudy.com> Message-ID: <20050422081550.GU22996@solar.trillke.net> Hi Ian, On Thu, Apr 21, 2005 at 22:38 -0500, Ian Bicking wrote: > Well, I'm doing a big renaming, and the errors from bad imports are a > bit confusing. It seems whenever there's an import error, the module > with the error still gets loaded, but is empty. This is a bit confusing > to debug. Well, this isn't true in the test_* module, but in the > modules they import. are you using a recent version of the py lib? py.test at least shouldn't trigger any special imports anymore but uses the underlying import mechanism. Could you post one of the offending tracebacks? thanks, holger From hpk at trillke.net Fri Apr 22 22:12:00 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 22 Apr 2005 22:12:00 +0200 Subject: [py-dev] distutils question Message-ID: <20050422201159.GI22996@solar.trillke.net> Hi folks, i have written simple distutils scripts before but i am now looking into writing a setup.py (and helpers) for the py lib. Thing is i would like to just say: please install the full package including any and all of its files (yes that means including '.svn' files, which you may disagree with but let's not argue about that right now). Is there an easy way to do it or do i have to hack things? I somehow can't believe that i have to more or less trick distutils into just installing a full directory tree, (compiling any pyc file in it as a bonus :-). Oh, and please only suggest things that work on python 2.2.x onwards if possible. any help and suggestions welcome, holger From ianb at colorstudy.com Sat Apr 23 03:40:09 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 22 Apr 2005 20:40:09 -0500 Subject: [py-dev] distutils question In-Reply-To: <20050422201159.GI22996@solar.trillke.net> References: <20050422201159.GI22996@solar.trillke.net> Message-ID: <4269A779.1000109@colorstudy.com> holger krekel wrote: > i have written simple distutils scripts before but i am now > looking into writing a setup.py (and helpers) for the py lib. > > Thing is i would like to just say: please install the full > package including any and all of its files (yes that means > including '.svn' files, which you may disagree with but let's > not argue about that right now). Is there an easy way to do > it or do i have to hack things? > > I somehow can't believe that i have to more or less trick > distutils into just installing a full directory tree, > (compiling any pyc file in it as a bonus :-). > > Oh, and please only suggest things that work on > python 2.2.x onwards if possible. I might suggest Python eggs, though unfortunately they are Python 2.3 and onward: http://peak.telecommunity.com/DevCenter/PythonEggs -- I haven't had a chance to use them, but it's next on my list. Otherwise, I believe you have to enumerate all the files to be installed (besides .py files in the enumerated packages), which you can do automatically if there's a lot (os.listdir and the like). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Sat Apr 23 03:45:27 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 22 Apr 2005 20:45:27 -0500 Subject: [py-dev] Import errors In-Reply-To: <20050422081550.GU22996@solar.trillke.net> References: <4268719B.2000103@colorstudy.com> <20050422081550.GU22996@solar.trillke.net> Message-ID: <4269A8B7.7080908@colorstudy.com> holger krekel wrote: > Hi Ian, > > On Thu, Apr 21, 2005 at 22:38 -0500, Ian Bicking wrote: > >>Well, I'm doing a big renaming, and the errors from bad imports are a >>bit confusing. It seems whenever there's an import error, the module >>with the error still gets loaded, but is empty. This is a bit confusing >>to debug. Well, this isn't true in the test_* module, but in the >>modules they import. > > > are you using a recent version of the py lib? > > py.test at least shouldn't trigger any special imports > anymore but uses the underlying import mechanism. > > Could you post one of the offending tracebacks? To repeat: $ svn co http://codespeak.net/svn/user/ianb/py_test_import_failure A py_test_import_failure/import_fails.py A py_test_import_failure/test_import_fail.py Checked out revision 11048. $ cd py_test_import_failure/ py_test_import_failure$ py.test inserting into sys.path: /home/ianb/py/dist ============================ test process starts ============================ testing-mode: inprocess executable: /usr/bin/python (2.3.5-final-0) py_test_import_failure/test_import_fail.py[-1] F _____________________________________________________________________________ def test_this(): E assert import_fails.a == 1 > AttributeError: 'module' object has no attribute 'a' [/home/ianb/t/py_test_import_failure/test_import_fail.py:4] [modulepath: test_this] _____________________________________________________________________________ ================= tests finished: 1 failed in 0.02 seconds ================= -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Sat Apr 23 03:49:54 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 22 Apr 2005 20:49:54 -0500 Subject: [py-dev] distutils question In-Reply-To: <4269A779.1000109@colorstudy.com> References: <20050422201159.GI22996@solar.trillke.net> <4269A779.1000109@colorstudy.com> Message-ID: <4269A9C2.7010200@colorstudy.com> Ian Bicking wrote: > holger krekel wrote: > >> i have written simple distutils scripts before but i am now >> looking into writing a setup.py (and helpers) for the py lib. >> >> Thing is i would like to just say: please install the full >> package including any and all of its files (yes that means >> including '.svn' files, which you may disagree with but let's >> not argue about that right now). Is there an easy way to do >> it or do i have to hack things? >> I somehow can't believe that i have to more or less trick >> distutils into just installing a full directory tree, >> (compiling any pyc file in it as a bonus :-). >> Oh, and please only suggest things that work on python 2.2.x onwards >> if possible. > > > I might suggest Python eggs, though unfortunately they are Python 2.3 > and onward: http://peak.telecommunity.com/DevCenter/PythonEggs -- I > haven't had a chance to use them, but it's next on my list. Alternately, I don't know if setuptools requires Python 2.3, and it may be easier for you to use. setuptools is what's used to build Python eggs. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From bob at redivi.com Sat Apr 23 04:11:05 2005 From: bob at redivi.com (Bob Ippolito) Date: Fri, 22 Apr 2005 22:11:05 -0400 Subject: [py-dev] distutils question In-Reply-To: <4269A9C2.7010200@colorstudy.com> References: <20050422201159.GI22996@solar.trillke.net> <4269A779.1000109@colorstudy.com> <4269A9C2.7010200@colorstudy.com> Message-ID: On Apr 22, 2005, at 9:49 PM, Ian Bicking wrote: > Ian Bicking wrote: >> holger krekel wrote: >>> i have written simple distutils scripts before but i am now >>> looking into writing a setup.py (and helpers) for the py lib. >>> >>> Thing is i would like to just say: please install the full >>> package including any and all of its files (yes that means >>> including '.svn' files, which you may disagree with but let's >>> not argue about that right now). Is there an easy way to do >>> it or do i have to hack things? I somehow can't believe that i have >>> to more or less trick >>> distutils into just installing a full directory tree, >>> (compiling any pyc file in it as a bonus :-). Oh, and please only >>> suggest things that work on python 2.2.x onwards if possible. >> I might suggest Python eggs, though unfortunately they are Python 2.3 >> and onward: http://peak.telecommunity.com/DevCenter/PythonEggs -- I >> haven't had a chance to use them, but it's next on my list. > > Alternately, I don't know if setuptools requires Python 2.3, and it > may be easier for you to use. setuptools is what's used to build > Python eggs. setuptools should work in 2.2 .. it used to anyway. -bob From hpk at trillke.net Sat Apr 23 12:40:15 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 23 Apr 2005 12:40:15 +0200 Subject: [py-dev] Import errors In-Reply-To: <4269A8B7.7080908@colorstudy.com> References: <4268719B.2000103@colorstudy.com> <20050422081550.GU22996@solar.trillke.net> <4269A8B7.7080908@colorstudy.com> Message-ID: <20050423104015.GL22996@solar.trillke.net> On Fri, Apr 22, 2005 at 20:45 -0500, Ian Bicking wrote: > >Could you post one of the offending tracebacks? > > To repeat: > > $ svn co http://codespeak.net/svn/user/ianb/py_test_import_failure > A py_test_import_failure/import_fails.py > A py_test_import_failure/test_import_fail.py > Checked out revision 11048. > $ cd py_test_import_failure/ > py_test_import_failure$ py.test > inserting into sys.path: /home/ianb/py/dist > ============================ test process starts > ============================ > testing-mode: inprocess > executable: /usr/bin/python (2.3.5-final-0) > > py_test_import_failure/test_import_fail.py[-1] F > _____________________________________________________________________________ > > def test_this(): > E assert import_fails.a == 1 > > AttributeError: 'module' object has no attribute 'a' > > [/home/ianb/t/py_test_import_failure/test_import_fail.py:4] > [modulepath: test_this] > _____________________________________________________________________________ > ================= tests finished: 1 failed in 0.02 seconds ================= OK, this one is actually related to a different problem, namely that the "[]" number in: py_test_import_failure/test_import_fail.py[-1] F it basically means that we try early to look forward to determine the number of tests. This is where the import fails. Later when the module is really run for error reporting the imports don't fail anymore (at least on python <2.4). Funny, that our custom import hook didn't have this problem because it could emulate python2.4 behaviour (deleting a failed import from sys.modules recursively). There are many paths that lead to hell starting from a simple import. Anyway, i fixed py.test to a) get around the problem and b) report a nice traceback. Thanks for pointing it out and providing an example! Actually i named the test that i added after you :-) cheers, holger From hpk at trillke.net Sun Apr 24 01:05:43 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 24 Apr 2005 01:05:43 +0200 Subject: [py-dev] towards a 0.8.0 release! Message-ID: <20050423230543.GO22996@solar.trillke.net> Hi folks, a few notes and questions regarding the to-be-released py lib, named '0.8.0': - there now is a setup.py in the 'svn/py/dist' directory which should work at least on unix-ish systems. Note that the "setup install" command will basically install the whole tree of the py lib including all svn-control files. This means that you can go to '../python-2.X/site-packages/py' and issue 'svn up'. It also means that when you fix things, you can run 'svn diff' there to create a patch. Or, if you have a codespeak account you can branch easily to test some idea out etc.pp. - the main missing piece is testing things much more on windows (speaking of which: has anyone recently used py.test on win32 and could try out: python py/bin/py.test py and possibly python setup.py install that would be great. I am sure that at least the installation of 'py.test' as a script will not work and i have not clue how to do it myself. Brian Dorsey suggested in private conversation to write an Immo binary Installer. - when 0.8.0 comes out, i basically want the directory release layout to look like this: http://codespeak.net/py/current/doc/releasescheme.html this means in particular: * svn/py/trunk will then hold head development * svn/py/branch/0.8 will be the ongoing release branch * svn/py/tag/0.8.x will be a tagged release * svn/py/dist will always hold the newest release and will not be writeable/commitable most of the time (i'll also prevent commits to 'tag/', i guess) - until 1.0 there will be _no_ particular release guarantees but i will be trying to follow and refine: http://codespeak.net/py/current/doc/why_py.html#upcoming-release-policy-api-guarantees - i'd like to have 1.0 latest at EuroPython end of June feedback & help welcome, holger From arigo at tunes.org Sun Apr 24 14:23:11 2005 From: arigo at tunes.org (Armin Rigo) Date: Sun, 24 Apr 2005 13:23:11 +0100 Subject: [py-dev] greenlets on amd64 (x86_64) Message-ID: <20050424122311.GA23725@vicky.ecs.soton.ac.uk> Hi, Some time ago, Hye-Shik Chang posted a patch for Stackless to support switching on amd64, on the stackless mailing list. I have put his switch_amd64_unix.h into py's greenlets and gave it a bit of (successful) testing. Enjoy! A bientot, Armin From briandorsey at gmail.com Sun Apr 24 20:50:41 2005 From: briandorsey at gmail.com (Brian Dorsey) Date: Sun, 24 Apr 2005 11:50:41 -0700 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <20050423230543.GO22996@solar.trillke.net> References: <20050423230543.GO22996@solar.trillke.net> Message-ID: <66e877b7050424115044331a46@mail.gmail.com> On 23/04/05, holger krekel wrote: > Hi folks, > - the main missing piece is testing things much more > on windows (speaking of which: has anyone recently > used py.test on win32 and could try out: > > python py/bin/py.test py This mostly passes for me on my home Windows XP machine. Only two of the tests give errors, but they give lots of them. py/path/svn/testing/test_urlcommand.py[73] FFF.F.........FFF..FFFFFFF...FFFFFFFF F.FFF......FFFFFFF.FFF.FFFFFFFF.FFFF py/path/svn/testing/test_wccommand.py[85] FFFFFFFFFFF...FFFFF.......F........FFF ...F I'm seeing lots of errors like this: Error: [None]: svn: Unrecognized URL scheme ''file://c%3A%5Cdocume%7E1%5Cbrian%5Clocals%7E1%5Ctemp%5Cpytest-0%5Crepo/otherdir'' So I may have an old or different version of svn installed. I'll see if I can track down the failures. That said... I'm probably not a normal windows user... I doubt most windows users even have svn in their path. ;) > and possibly > > python setup.py install I'll try to look at this soon... but it may not happen... life has gotten busy for me again. :) > that would be great. I am sure that at least > the installation of 'py.test' as a script will > not work and i have not clue how to do it myself. > Brian Dorsey suggested in private conversation > to write an Immo binary Installer. I was thinking of Inno Setup: http://www.jrsoftware.org/isinfo.php ... that said... if we can get the standard distutils version working, that's probably the best? This is a bit new to me as well... I've never really distributed python modules for windows, just applications. (That said... the module installers I have used which also have scripts never seem to just work for me. :) Take care, -Brian From p.f.moore at gmail.com Sun Apr 24 21:22:02 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 24 Apr 2005 20:22:02 +0100 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <66e877b7050424115044331a46@mail.gmail.com> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> Message-ID: <79990c6b050424122248a98c1@mail.gmail.com> On 4/24/05, Brian Dorsey wrote: > I was thinking of Inno Setup: http://www.jrsoftware.org/isinfo.php > ... that said... if we can get the standard distutils version working, > that's probably the best? This is a bit new to me as well... I've > never really distributed python modules for windows, just > applications. (That said... the module installers I have used which > also have scripts never seem to just work for me. :) I'd very strongly recommend sticking with a standard setup.py/bdist_wininst installer for Windows. It's what nearly every other package uses. I just did a quick test of bdist_wininst and it failed to tidy up properly, I think largely because it couldn't remove the Subversion directories. I strongly disagree with the idea of including the Subversion control stuff in the installed copy, in any case. In my view, you shouldn't be doing "svn up" in your site-packages tree, rather you should be doing it in your build tree, then building a new installer and running it. This keeps the registry of installed packages up to date. Of course, on Unix where there is no uninstall facility (if you puse setup.py install, at least) this is less of an issue, but it is true on Windows. Is there any chance of changing the decision to include the .svn directories - on Windows, if not elsewhere? Other issues with setup.py - it doesn't seem to build C extensions - greenlets, specifically, but I notice that popen5 (_subprocess.c) is included as well - is this right? If so, I'd expect it to be excluded from Python 2.4 and above builds (which is problematic as long as there are no C extensions included, as in that case the installer is version-independent). Sorry, but I'm not able to go much further with this at the moment. I hope the above comments are of some help in any case. Paul. From hpk at trillke.net Sun Apr 24 22:07:21 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 24 Apr 2005 22:07:21 +0200 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <79990c6b050424122248a98c1@mail.gmail.com> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <79990c6b050424122248a98c1@mail.gmail.com> Message-ID: <20050424200721.GA22996@solar.trillke.net> Hi Paul, On Sun, Apr 24, 2005 at 20:22 +0100, Paul Moore wrote: > I just did a quick test of bdist_wininst and it failed to tidy up > properly, I think largely because it couldn't remove the Subversion > directories. What exactly failed? The .svn-directories are installled as data files so i am curious why it doesn't work. And does that mean that the installation and i invoking 'py.test' worked? > I strongly disagree with the idea of including the > Subversion control stuff in the installed copy, in any case. In my > view, you shouldn't be doing "svn up" in your site-packages tree, .. it's also 'svn info' and 'svn diff' that is useful IMO ... > rather you should be doing it in your build tree, then building a new > installer and running it. That should of course work but it's not neccessarily excluding the other approach. > This keeps the registry of installed packages up to date. I guess you are refering to the Windows registry, right? What information is in there exactly? Like a list of packages with versions and the list of installed files or some such? > Of course, on Unix where there is no uninstall facility (if > you puse setup.py install, at least) this is less of an > issue, but it is true on Windows. > > Is there any chance of changing the decision to include the .svn > directories - on Windows, if not elsewhere? Yes, there are chances. I'd like to understand the problems better, though. Doing things too differently on windows and unix is certainly undesirable. > Other issues with setup.py - it doesn't seem to build C extensions - > greenlets, specifically, Sorry, C-extensions are not built at all currently. > but I notice that popen5 (_subprocess.c) is included as well > - is this right? Yes, the py lib aims to run on python 2.2 upwards. > If so, I'd expect it to be excluded from Python 2.4 and > above builds (which is problematic as long as there are no C > extensions included, as in that case the installer is > version-independent). Indeed, greenlets & subprocess will probably make the installer version dependent, i am afraid. Is it black magic to build version dependent installers or is it rather straight forward? > Sorry, but I'm not able to go much further with this at the moment. I > hope the above comments are of some help in any case. Yes, thanks a lot! holger From hpk at trillke.net Sun Apr 24 22:10:25 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 24 Apr 2005 22:10:25 +0200 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <66e877b7050424115044331a46@mail.gmail.com> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> Message-ID: <20050424201025.GB22996@solar.trillke.net> Hi Brian! On Sun, Apr 24, 2005 at 11:50 -0700, Brian Dorsey wrote: > On 23/04/05, holger krekel wrote: > > Hi folks, > > > - the main missing piece is testing things much more > > on windows (speaking of which: has anyone recently > > used py.test on win32 and could try out: > > > > python py/bin/py.test py > > This mostly passes for me on my home Windows XP machine. Only two of > the tests give errors, but they give lots of them. > > py/path/svn/testing/test_urlcommand.py[73] FFF.F.........FFF..FFFFFFF...FFFFFFFF > F.FFF......FFFFFFF.FFF.FFFFFFFF.FFFF > py/path/svn/testing/test_wccommand.py[85] FFFFFFFFFFF...FFFFF.......F........FFF > ...F > > I'm seeing lots of errors like this: > Error: [None]: svn: Unrecognized URL scheme > ''file://c%3A%5Cdocume%7E1%5Cbrian%5Clocals%7E1%5Ctemp%5Cpytest-0%5Crepo/otherdir'' > > So I may have an old or different version of svn installed. Quite likely, it's a problem of command line parsing / escaping things properly. > I'll see if I can track down the failures. > > That said... I'm probably not a normal windows user... I doubt most > windows users even have svn in their path. ;) Hum, i see. What is a proper way to get at the location of such a binary on win32? cheers & thanks, holger From jum at anubis.han.de Sun Apr 24 23:07:34 2005 From: jum at anubis.han.de (Jens-Uwe Mager) Date: Sun, 24 Apr 2005 23:07:34 +0200 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <20050424201025.GB22996@solar.trillke.net> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <20050424201025.GB22996@solar.trillke.net> Message-ID: <20050424210733.GG924@anubis.han.de> On Sun, Apr 24, 2005 at 22:10 +0200, holger krekel wrote: > On Sun, Apr 24, 2005 at 11:50 -0700, Brian Dorsey wrote: > > > > That said... I'm probably not a normal windows user... I doubt most > > windows users even have svn in their path. ;) > > Hum, i see. What is a proper way to get at the location > of such a binary on win32? The canonical way would be to look for certain popular ssh implementations in the registry about their install location. But accessing the registry requires the win32 python extensions, which are optional as well. And then the binary might not even be named ssh, for example with the popular PuTTY package the command line remote exec utility is named plink.exe. -- Jens-Uwe Mager -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available URL: From p.f.moore at gmail.com Sun Apr 24 23:28:09 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 24 Apr 2005 22:28:09 +0100 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <20050424201025.GB22996@solar.trillke.net> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <20050424201025.GB22996@solar.trillke.net> Message-ID: <79990c6b05042414285a8970ee@mail.gmail.com> On 4/24/05, holger krekel wrote: > Hum, i see. What is a proper way to get at the location > of such a binary on win32? Assume it's in PATH. If it isn't, there's no "proper" way. Actually, don't even assume that. I might well have a "svn.bat" in my path, which executes svn.exe from a particular directory. Just assume the command "svn" will work, and handle exceptions gracefully. (Is my hardline attitude becoming clear yet? :-)) Paul. PS I'm not actually sure this is the real issue here. That file: URL quoted looks distinctly odd - possibly legal, but definitely odd. Windows programs have "interesting" views on valid forms of file URLs (I don't what to go into details here, I'd end up defending a complex mess as being pragmatically the right thing...) PPS FWIW, I got similar patterns of failures in the two tests quoted, but no mention of odd URLs. I'm not going to post the error tracebacks, as there were a lot (the second was 868 lines) and I have no time to even try to help diagnose the issue. I'd also point out that when I ran the full test, the execnet stuff seemed to hang, but I don't know why. From bob at redivi.com Sun Apr 24 23:30:23 2005 From: bob at redivi.com (Bob Ippolito) Date: Sun, 24 Apr 2005 17:30:23 -0400 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <79990c6b05042414285a8970ee@mail.gmail.com> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <20050424201025.GB22996@solar.trillke.net> <79990c6b05042414285a8970ee@mail.gmail.com> Message-ID: <6aa69c2b343ce035e54ace9d53b5b323@redivi.com> On Apr 24, 2005, at 5:28 PM, Paul Moore wrote: > On 4/24/05, holger krekel wrote: >> Hum, i see. What is a proper way to get at the location >> of such a binary on win32? > > Assume it's in PATH. If it isn't, there's no "proper" way. Actually, > don't even assume that. I might well have a "svn.bat" in my path, > which executes svn.exe from a particular directory. Just assume the > command "svn" will work, and handle exceptions gracefully. Why not look in the registry for hints that the win32 installer for svn was used, and if so, use that? -bob From hpk at trillke.net Sun Apr 24 23:48:54 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 24 Apr 2005 23:48:54 +0200 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <79990c6b050424135321873c78@mail.gmail.com> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <79990c6b050424122248a98c1@mail.gmail.com> <20050424200721.GA22996@solar.trillke.net> <79990c6b050424135321873c78@mail.gmail.com> Message-ID: <20050424214854.GC22996@solar.trillke.net> Hi Paul, trying to strip it down a bit ... On Sun, Apr 24, 2005 at 21:53 +0100, Paul Moore wrote: > On 4/24/05, holger krekel wrote: > > And does that mean that the installation and i invoking > > 'py.test' worked? > > Of course not :-) as the file was installed as "py.test" and not as > some circumlocution involving a .bat file, or as py.test.py, to be run > with that name. OK, i had the faint and irrational hope that things passed as 'scripts' to setup() would maybe install them appropriately on windows (which shouldn't be impossible to achieve, should it?). > On the other hand, copying the "py.test" script to somewhere else > (somewhere on my PATH) and renaming it as, for example, py.test.py, > fails with ... Yes, I found it. It's the _findpy.py module > again. I stand by my comments from a while back, that the > existence of this module is a symptom of a problem. To find > the "py" module, you should be doing "import py"!!! It does fall back to 'import py' these days! Btw, I haven't of heart of problems regarindg _findpy (once you have it properly installed) and it does definitely have advantages at least being a developer, working with different versions. > ... > > [Meta-comment here - I still get the feeling that there's "too much > magic" going on. Please also understand that i don't currently have access to a win32 platform. > py.test script, and move it anywhere I want, and rename it as I like, > and have it still work. That doesn't happen, and that counts as > "magic" to me.] Maybe the _findpy logic should be folded into py.test itself so that it becomes self contained and can easily work with a site-packages installed py lib. > > > I strongly disagree with the idea of including the > > > Subversion control stuff in the installed copy, in any case. In my > > > view, you shouldn't be doing "svn up" in your site-packages tree, > > > > .. it's also 'svn info' and 'svn diff' that is useful IMO ... > > Again, not from the installed copy. If it matters, make the version > include the Subversion revision number, by all means, but that's all. > Unix practice may be different, but on Windows, I'd consider it very > bad practice to be working in a directory under "Python24" - ever. ok, i guess i will be dropping it. In fact i put new "0.8.0-alpha1" installers for unix & windows at http://codespeak.net/download/py They don't contain the '.svn' directories anymore but i am not sure that the windows installer really contains all the files ... Generally, i have to admit that i am just frustrated at the lack of versioning/dependency handling by Python's distribution mechanisms. It has bitten me more than once and i wanted to try out the 'have it svn ready just in case' to get some anchor in it. > ... > ("C:\Apps\Python24\Removepy.exe" -u "C:\Apps\Python24\py-wininst.log") > - the latter is built by the bdist_wininst installer and contains a > listing of all the files and directories installed. Doing something > like "svn up" from within the site-packages directory would totally > break this uninstaller. Side note: i like the notion of having installed applications completely contained in one directory. That also makes uninstallation trivial. > Perhaps you could explain why you *don't* have an issue with using svn > on installed files? There simply isn't any problem to note from my side. (i am running it on two production machines ...). > > Sorry, C-extensions are not built at all currently. > > Fair enough. But when you do get it building C extensions, you'll get > installer support on Windows "for free" - if setup.py build works, > then setup.py bdist_wininst does too. It's pretty slick. I hope it's feasible to get a win32 build environment for 2.2, and 2.3 and 2.4 on one machine going (and without spending money for licenses ...). cheers, holger From briandorsey at gmail.com Mon Apr 25 01:52:27 2005 From: briandorsey at gmail.com (Brian Dorsey) Date: Sun, 24 Apr 2005 16:52:27 -0700 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <20050424201025.GB22996@solar.trillke.net> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <20050424201025.GB22996@solar.trillke.net> Message-ID: <66e877b7050424165234b68103@mail.gmail.com> On 24/04/05, holger krekel wrote: > > I'm seeing lots of errors like this: > > Error: [None]: svn: Unrecognized URL scheme > > ''file://c%3A%5Cdocume%7E1%5Cbrian%5Clocals%7E1%5Ctemp%5Cpytest-0%5Crepo/otherdir'' > > > > So I may have an old or different version of svn installed. > > Quite likely, it's a problem of command line parsing / escaping things properly. OK, I didn't have much time to look at this carefullly, but there is at least a problem with the escaping of quotes on the command line, and even after doing what I think should resolve that, it doesn't seem to like all the URL safety '%' in the path. I get things like this back from svn: D:\BrianDocuments\home\local\py-dist> svn ls "-v" file://c%3A%5Cdocume%7E1%5Cbri an%5Clocals%7E1%5Ctemp%5Cpytest-1%5Crepo/otherdir svn: Unable to open an ra_local session to URL svn: Local URL 'file://c%3A%5Cdocume%7E1%5Cbrian%5Clocals%7E1%5Ctemp%5Cpytest-1% 5Crepo/otherdir' contains unsupported hostname I've run out of time to play with this today (and, honestly, didn't get to spend that much time on it. ;) but, I'd like to present one option for the 0.8 release: The py.test funtionality is the most important to me, (and perhaps generally?) and it wouldn't bother me even the tiniest little bit if none of the C modules, SVN magic or anything except py.test didn't work on Windows for this release. ;) > > That said... I'm probably not a normal windows user... I doubt most > > windows users even have svn in their path. ;) > > Hum, i see. What is a proper way to get at the location > of such a binary on win32? I'm with Paul on this one. If it's not in the path, don't do svn stuff. If the user wants to do svn stuff, make them modify their path and point to the copy of svn that they want to use. (actually... I'll bet most windows users... even most python users probably don't have svn even installed.) Take care, -Brian From pedronis at strakt.com Mon Apr 25 02:18:14 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Mon, 25 Apr 2005 02:18:14 +0200 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <66e877b7050424165234b68103@mail.gmail.com> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <20050424201025.GB22996@solar.trillke.net> <66e877b7050424165234b68103@mail.gmail.com> Message-ID: <426C3746.8090705@strakt.com> Brian Dorsey wrote: > On 24/04/05, holger krekel wrote: > >>>I'm seeing lots of errors like this: >>>Error: [None]: svn: Unrecognized URL scheme >>>''file://c%3A%5Cdocume%7E1%5Cbrian%5Clocals%7E1%5Ctemp%5Cpytest-0%5Crepo/otherdir'' >>> >>>So I may have an old or different version of svn installed. >> >>Quite likely, it's a problem of command line parsing / escaping things properly. > > > OK, I didn't have much time to look at this carefullly, but there is > at least a problem with the escaping of quotes on the command line, > and even after doing what I think should resolve that, it doesn't seem > to like all the URL safety '%' in the path. I get things like this > back from svn: > D:\BrianDocuments\home\local\py-dist> svn ls "-v" file://c%3A%5Cdocume%7E1%5Cbri > an%5Clocals%7E1%5Ctemp%5Cpytest-1%5Crepo/otherdir > svn: Unable to open an ra_local session to URL > svn: Local URL 'file://c%3A%5Cdocume%7E1%5Cbrian%5Clocals%7E1%5Ctemp%5Cpytest-1% > 5Crepo/otherdir' contains unsupported hostname > well the first thing I notice is file:// instead of file:///, maybe it works on Unix with two slashes but the file: protocol really needs three, I remember on windows this being quite important because of the drive thing. Or file://localhost/. See also http://svnbook.red-bean.com/en/1.0/ch02s03.html#svn-ch-2-table-1 And you probably don't want back-slashes even % encoded, but just slashes. From pedronis at strakt.com Mon Apr 25 02:39:42 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Mon, 25 Apr 2005 02:39:42 +0200 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <426C3746.8090705@strakt.com> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <20050424201025.GB22996@solar.trillke.net> <66e877b7050424165234b68103@mail.gmail.com> <426C3746.8090705@strakt.com> Message-ID: <426C3C4E.4060004@strakt.com> Samuele Pedroni wrote: > > well the first thing I notice is file:// instead of file:///, maybe it > works on Unix with two slashes but the file: protocol really needs > three, I remember on windows this being quite important because of the > drive thing. Or file://localhost/. > See also > > http://svnbook.red-bean.com/en/1.0/ch02s03.html#svn-ch-2-table-1 > > > And you probably don't want back-slashes even % encoded, but just slashes. > indeed reading the relevant specs % encoding can be used inside path segments (that means what is between // slashes) but the segments themself need to separated by plain unencoded slashes. From ianb at colorstudy.com Mon Apr 25 05:18:52 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 24 Apr 2005 22:18:52 -0500 Subject: [py-dev] print statements during collect Message-ID: <426C619C.7010108@colorstudy.com> When I have print statements during the collect phase, they seem to be lost forever (unless I turn off stdout capture). I think it would be best if, in the case of any error, all the printout during collection was also shown (probably distinguished from prints during test runs). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From florian.proff.schulze at gmx.net Mon Apr 25 12:18:18 2005 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Mon, 25 Apr 2005 12:18:18 +0200 Subject: [py-dev] towards a 0.8.0 release! In-Reply-To: <20050424200721.GA22996@solar.trillke.net> References: <20050423230543.GO22996@solar.trillke.net> <66e877b7050424115044331a46@mail.gmail.com> <79990c6b050424122248a98c1@mail.gmail.com> <20050424200721.GA22996@solar.trillke.net> Message-ID: On Sun, 24 Apr 2005 22:07:21 +0200, holger krekel wrote: > Indeed, greenlets & subprocess will probably make the > installer version dependent, i am afraid. Is it black magic > to build version dependent installers or is it rather straight > forward? As soon as you compile C extensions, you will get an installer which is only for the python version used to build the installer. So to build an python 2.3 installer you run it with python 2.3 etc. Regards, Florian Schulze From ianb at colorstudy.com Tue Apr 26 04:29:19 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 25 Apr 2005 21:29:19 -0500 Subject: [py-dev] doctests again Message-ID: <426DA77F.9000507@colorstudy.com> I'm just checking in... is there any preferred way to do doctests with py.test now? Holger mentioned something, but he hadn't integrated reporting yet or something. But fancy reporting or not, I'd like to run those doctests too... I can dig up that old recipe I posted a while ago, but maybe there's a better way to do it now. Does doctest.DocTestSuite() work okay? I haven't looked at the unittest compatibility since it was mentioned. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From hpk at trillke.net Tue Apr 26 10:56:00 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 26 Apr 2005 10:56:00 +0200 Subject: [py-dev] doctests again In-Reply-To: <426DA77F.9000507@colorstudy.com> References: <426DA77F.9000507@colorstudy.com> Message-ID: <20050426085600.GR22996@solar.trillke.net> Hi Ian, On Mon, Apr 25, 2005 at 21:29 -0500, Ian Bicking wrote: > I'm just checking in... is there any preferred way to do doctests with > py.test now? No, there is no prefered way yet. To properly integrate it into py.test I still need to look up the exact mechanics of doctests and solve two issues i think: - how to integrate them into the collecting process - how to report them (the current report logic has a two unittest-specific view with respect to reports about failing tests) > Holger mentioned something, but he hadn't integrated > reporting yet or something. But fancy reporting or not, I'd like to run > those doctests too... > I can dig up that old recipe I posted a while ago, > but maybe there's a better way to do it now. Your old recipe would not work unmodified because there is no 'extpy' anymore. Actually i'd like to write some documentation about the (refactored) collecting process and think about how to integrate them into the collecting process. > Does doctest.DocTestSuite() work okay? Sorry, I am not exactly sure what you are refering to here. > I haven't looked at the unittest compatibility since it was > mentioned. There is no general unittest compatibility, except for running certain regression tests via py.test on PyPy. At some point i want to factor out the PyPy hacks into a generally useful compatible runner of traditional unittests but that's not too high on my priority list. cheers, holger From michele.simionato at gmail.com Tue Apr 26 19:59:11 2005 From: michele.simionato at gmail.com (Michele Simionato) Date: Tue, 26 Apr 2005 13:59:11 -0400 Subject: [py-dev] my first post to the list Message-ID: <4edc17eb050426105935f7e824@mail.gmail.com> Well, after seeing Armin presentation at the ACCU conference I have decided to keep a close eye on what it is going on at codespeak.net ;) He gave a couple of pretty convincing talks. Personally, I am more interested in the py library, since it works with current Python now, whereas the PyPy project is a more long term one. So today I spent some time playing with greenlets. As an exercise, I was trying to implement generators using greenlets (I know there is an example on site, but I wanted to do it on my own) and I am a bit confused. I wanted to write the equivalent of this: def _gen123(): yield 1 yield 2 yield 3 for i in _gen123(): print i I thought this should work: from py.magic import greenlet @greenlet def main(): while True: try: print "switching ..." print gen123.switch() except StopIteration: break @greenlet def gen123(): print "gen123 start ..." greenlet.getcurrent().parent.switch(1) greenlet.getcurrent().parent.switch(2) greenlet.getcurrent().parent.switch(3) raise StopIteration main.switch() print "Done" However, it does not work as I would expect, the controls goes immediately back to the toplevel. Any hint about what I am doing wrong? Michele Simionato From michele.simionato at gmail.com Wed Apr 27 10:55:11 2005 From: michele.simionato at gmail.com (Michele Simionato) Date: Wed, 27 Apr 2005 04:55:11 -0400 Subject: [py-dev] Re: my first post to the list In-Reply-To: <4edc17eb050426105935f7e824@mail.gmail.com> References: <4edc17eb050426105935f7e824@mail.gmail.com> Message-ID: <4edc17eb050427015517923b07@mail.gmail.com> On 4/26/05, Michele Simionato wrote: > As an exercise, I was trying to implement generators using > greenlets (I know there is an example on site, but I wanted > to do it on my own) and I am a bit confused. For the record, I have solved the confusion on my own. I misunderstood what the parent greenlet is. I had the wrong idea that the parent greenlet was the calling greenlet, whereas it is just the greenlet where the child greenlet is *defined*, not called. So my example works just by setting the right parent by hand: gen123.parent = main I see that the same trick is used in test_generators.py. Still thinking to practical applications of the idea, but I believe I understand the basics now. Michele From arigo at tunes.org Thu Apr 28 11:21:42 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Apr 2005 10:21:42 +0100 Subject: [py-dev] my first post to the list In-Reply-To: <4edc17eb050426105935f7e824@mail.gmail.com> References: <4edc17eb050426105935f7e824@mail.gmail.com> Message-ID: <20050428092142.GA7369@vicky.ecs.soton.ac.uk> Hi all, On Tue, Apr 26, 2005 at 01:59:11PM -0400, Michele Simionato wrote: > Well, after seeing Armin presentation at the ACCU conference > I have decided to keep a close eye on what it is going on > at codespeak.net ;) For the records, as Michele asked me, I've put the greenlet example I gave in the lightning talk into the greenlet documentation: http://codespeak.net/py/current/doc/greenlet.html#example Armin From michele.simionato at gmail.com Thu Apr 28 11:33:10 2005 From: michele.simionato at gmail.com (Michele Simionato) Date: Thu, 28 Apr 2005 05:33:10 -0400 Subject: [py-dev] my first post to the list In-Reply-To: <20050428092142.GA7369@vicky.ecs.soton.ac.uk> References: <4edc17eb050426105935f7e824@mail.gmail.com> <20050428092142.GA7369@vicky.ecs.soton.ac.uk> Message-ID: <4edc17eb050428023316ab7f83@mail.gmail.com> On 4/28/05, Armin Rigo wrote: > For the records, as Michele asked me, I've put the greenlet example I gave in > the lightning talk into the greenlet documentation: > > http://codespeak.net/py/current/doc/greenlet.html#example I think the example could be improved a bit. We have def process_commands(*args): while True: line = '' while not line.endswith('\n'): line += read_next_char() if line == 'quit\n': print "are you sure?" if read_next_char() != 'y': continue # ignore the command process_command(line) but the *args are ignored and 'process_command' looks too similar to process_commands; also the continue line looks strange. Why not just def show_entered_commands(): while True: line = '' while not line.endswith('\n'): line += read_next_char() print line if line == 'quit\n': print "are you sure?" if read_next_char() == 'y': break ? Anyway, this is a pretty cool example of what you can do with greenlets. Michele From michele.simionato at gmail.com Fri Apr 29 10:58:54 2005 From: michele.simionato at gmail.com (Michele Simionato) Date: Fri, 29 Apr 2005 04:58:54 -0400 Subject: [py-dev] my first post to the list In-Reply-To: <4edc17eb050428023316ab7f83@mail.gmail.com> References: <4edc17eb050426105935f7e824@mail.gmail.com> <20050428092142.GA7369@vicky.ecs.soton.ac.uk> <4edc17eb050428023316ab7f83@mail.gmail.com> Message-ID: <4edc17eb05042901585d7c2225@mail.gmail.com> BTW, I was continuing my experiments with greenlets and I seems that I cannot pickle a greenlet, I cannot copy it reliably, and I cannot restart it. IOW, apparently there is no way to save an execution context and restart it later. Am I correct? These limitations should be documented. Michele Simionato From hpk at trillke.net Sat Apr 30 09:48:52 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 30 Apr 2005 09:48:52 +0200 Subject: [py-dev] capturing / doctests / py lib release Message-ID: <20050430074852.GX22996@solar.trillke.net> Hi py-dev, i am wondering if anybody would mind (or endorse) not catching stderr by default but only stdout? There would be a switch ('-e' or something) that would enable capturing of stderr as it is now. Or should there just be a switch to turn off stdout/stderr capturing separately but keep the defaults to catch both? On a side note, i am probably not going to release the py lib before end of May because of the missing fleshing out of win32 support. Nobody, including me, has the time and will so far to fix things about it. In case you haven't noticed, i am looking for a co-developer familiar with the win32 platform(s). You get full commit rights and strong influence on how things are done on win32 and the lib in general in return, apart from fame and glory. Generalizing this: if any of you wants to help drive the py lib please feel free to take things over. Btw, i am very happy that Jan Balster took over the development of a tkinter frontend ("py.test --tkinter") which looks very promising already. If you have suggestions or improvements for this frontend please post to this list, Jan would be happy to get feedback. I do, however, intend to work on doctest integration real soon now. Mind you, this is a free software programmer saying this but then one, who also is contractually bound by the European Union to get certain py.test related features working for PyPy so bets are not all off. I consider the "how to collect doctests" issues still the primary blocker design question. cheers, holger From michele.simionato at gmail.com Sat Apr 30 10:36:55 2005 From: michele.simionato at gmail.com (Michele Simionato) Date: Sat, 30 Apr 2005 04:36:55 -0400 Subject: [py-dev] capturing / doctests / py lib release In-Reply-To: <20050430074852.GX22996@solar.trillke.net> References: <20050430074852.GX22996@solar.trillke.net> Message-ID: <4edc17eb05043001364ac24e47@mail.gmail.com> On 4/30/05, holger krekel wrote: > I consider the "how to collect doctests" > issues still the primary blocker design question. I have some utility to do that (in the category "it works for me"). Given a package, it walks recursively on all submodules and subpackages, retrieving docstrings and executing them. At the end it returns a simple summary. Originally, I wanted to convert those doctests to unittests (it is actually trivial) but I did not need that, so I thought "why bother?". Also, very often I have doctests in external files, and I would like to collect them too. Life would be simpler however if there was a set of common conventions (i.e. doctest files have a specific extension, they go in a specific directory) etc. Does somebody know how they do that in Zope3 ? They for sure have *lots* of doctests. Michele Simionato