From ianb at colorstudy.com Fri Oct 1 03:24:43 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 30 Sep 2004 20:24:43 -0500 Subject: [py-dev] utest, development and discussion In-Reply-To: <20040930071527.GK19356@solar.trillke.net> References: <415672F3.9060808@colorstudy.com> <200409260821.i8Q8LqOU032224@ratthing-b246.strakt.com> <415B5775.3040402@colorstudy.com> <20040930071527.GK19356@solar.trillke.net> Message-ID: <415CB1DB.2020007@colorstudy.com> holger krekel wrote: >>Since std (I guess to be named py) isn't under pypy, I assume >>documentation should go in std/trunk/doc? Can it be set up that this is >>also turned into HTML? > > > yes, this is certainly possible although some of the html generation > is loosely tied to pypy. > > however, i want std/py is to be the first project which > will be hosted by 'trac' on the new codespeak setup. Will the documentation still be in ReST, or will it use another Wiki-ish markup? I was going to try to write some basic documentation (well, maybe this weekend), with whatever vague sense I have of how things work, and I just wanted to put it in the right place and format. And utest is a good test runner right now, I think documentation is the only thing that would keep people from using it. Well, a distutil setup or something equivalent that would also be helpful. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From lac at strakt.com Fri Oct 1 08:17:03 2004 From: lac at strakt.com (Laura Creighton) Date: Fri, 01 Oct 2004 08:17:03 +0200 Subject: [py-dev] utest, development and discussion In-Reply-To: Message from Ian Bicking of "Thu, 30 Sep 2004 20:24:43 CDT." <415CB1DB.2020007@colorstudy.com> References: <415672F3.9060808@colorstudy.com> <200409260821.i8Q8LqOU032224@ratthing-b246.strakt.com> <415B5775.3040402@colorstudy.com> <20040930071527.GK19356@solar.trillke.net> <415CB1DB.2020007@colorstudy.com> Message-ID: <200410010617.i916H37n015072@ratthing-b246.strakt.com> In a message of Thu, 30 Sep 2004 20:24:43 CDT, Ian Bicking writes: >holger krekel wrote: >>>Since std (I guess to be named py) isn't under pypy, I assume >>>documentation should go in std/trunk/doc? Can it be set up that this i > >>>also turned into HTML? >> >> >> yes, this is certainly possible although some of the html generation >> is loosely tied to pypy. >> >> however, i want std/py is to be the first project which >> will be hosted by 'trac' on the new codespeak setup. > >Will the documentation still be in ReST, or will it use another Wiki-ish >markup? I was going to try to write some basic documentation (well, >maybe this weekend), with whatever vague sense I have of how things >work, and I just wanted to put it in the right place and format. And >utest is a good test runner right now, I think documentation is the only >thing that would keep people from using it. Well, a distutil setup or >something equivalent that would also be helpful. > >-- >Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org I think there is nothing but ReST fans here. :-) Laura From hpk at trillke.net Fri Oct 1 09:28:11 2004 From: hpk at trillke.net (holger krekel) Date: Fri, 1 Oct 2004 09:28:11 +0200 Subject: [py-dev] utest, development and discussion In-Reply-To: <415CB1DB.2020007@colorstudy.com> References: <415672F3.9060808@colorstudy.com> <200409260821.i8Q8LqOU032224@ratthing-b246.strakt.com> <415B5775.3040402@colorstudy.com> <20040930071527.GK19356@solar.trillke.net> <415CB1DB.2020007@colorstudy.com> Message-ID: <20041001072811.GB7311@solar.trillke.net> [Ian Bicking Thu, Sep 30, 2004 at 08:24:43PM -0500] > holger krekel wrote: > >>Since std (I guess to be named py) isn't under pypy, I assume > >>documentation should go in std/trunk/doc? Can it be set up that this is > >>also turned into HTML? > > > > > >yes, this is certainly possible although some of the html generation > >is loosely tied to pypy. > > > >however, i want std/py is to be the first project which > >will be hosted by 'trac' on the new codespeak setup. > > Will the documentation still be in ReST, or will it use another Wiki-ish > markup? I was going to try to write some basic documentation (well, > maybe this weekend), with whatever vague sense I have of how things > work, and I just wanted to put it in the right place and format. And > utest is a good test runner right now, I think documentation is the only > thing that would keep people from using it. Well, a distutil setup or > something equivalent that would also be helpful. Right on all points. I guess that with test runner you also refer to the collection/running distinction. Actually ReST is probably good for "static" reliable documentation that comes with the package and wiki-ish stuff for all kinds of more dynamic/experimental/example stuff that has a high probability of changing. Of course, a simple distutils setup install wouldn't be a problem. But somehow i have this idea that i want to make it easy for people to upgrade to a newer version of the py lib and i haven't quite figured out how to do it. That how it is with me: i always try to fix other stuff before continuing to fix the current code at hand and this is a recursive process. Now maybe just more people need to start doing it or, more likely the right answer, i need to become more pragmatic :-) cheers, holger From hpk at trillke.net Fri Oct 1 18:44:12 2004 From: hpk at trillke.net (holger krekel) Date: Fri, 1 Oct 2004 18:44:12 +0200 Subject: [py-dev] utest thoughts In-Reply-To: <4158A934.4040800@colorstudy.com> References: <4158A934.4040800@colorstudy.com> Message-ID: <20041001164412.GN7311@solar.trillke.net> Hi Ian, hi everybody else, i have still not much time but i can't wait to reply anymore :-) please Armin, Jens-Uwe, everybody: if you feel anything is mis-represented or you have completely different ideas, or you just want to comment on it, go ahead! We may want to split the mail into different threads, though, if discussing it further. [Ian Bicking Mon, Sep 27, 2004 at 06:58:44PM -0500] > * Specify tests to run within a module. The only way to select a module > that I see now is by filename. Package name would also be nice. > Wildcards could also be useful, e.g., utest modulename.'*transaction*'. > I think regular expressions are unnecessarily complex. Maybe a > wildcard character other than * would be nice, to keep it from > conflicting with shell expansion. A setting, or an optional alternative > character? Maybe % (like in SQL). Yes, selecting test methods by (maybe wildcarded) expressions seems reasonable. However, i would like to first implement an "stateful testing" approach which would basically work like this: py.test --session ... runs all tests, some of which fail ... py.test --session ... runs only the previously failed tests ... repeat the last step until all tests pass, then run all tests --> go to step 1 I somehow think that this might satisfy many use cases which otherwise would require some wildcard based selection scheme. > * Data-driven tests, where the same code is tested with many different > sets of data. Naturally this is often done in a for loop, but it's > better if the data turns into multiple tests, each of which are > addressable. There's something called a "unit" in there, I think, that > relates to this...? But not the same thing as unittest; I think I saw > unittest compatibility code as well. > > Anyway, with unittest I could provide values to the __init__, creating > multiple tests that differed only according to data, but then the runner > became fairly useless. I'm hoping that be easier with utest. py.test consists of three main elements when "running" tests: the collector(s), the reporter (for producing all output), and the runner, driving the collector and the reporter. Collectors are completly decoupled from running tests. They are implemented (std/utest/collect.py) via a concept called "Restartable Iterators", thus the runner doesn't not need to know all the tests when starting to run them. I know that unittest.py approaches often just put all these three elements together and call it a "runner" but i think it blurs the lines and makes it very difficult to keep it extensible and customizable. Having said this, for having multiple data sets it's probably best to introduce a mechanism into the Module Collector to look for custom module-defined collectors which would seamlessly integrate into the collection process of Units. By implementing your own collection function you can yield many Units with different test data sets which are then executed/accounted for individually. > * Specifying an option to the runner that gets passed through to the > tests. It seems like the options are fixed now. I'd like to do > something like -Ddatabase=mysql. I can do this with environmental > variables now, but that's a little crude. It's easiest if it's just > generic, like -D for compilers, but of course it would be nicer if there > were specific options. Maybe this could be best achieved by > specializing utest and distributing my own runner with the project. I see the point. Doing this should be done by reexaminig the current 'config' mechanism. Or better: rewriting it alltogether as it is very much ad hoc/preliminary. There is a provision to have utest configuration files currently called utest.conf where you can at the moment only set some default values for command line options. with unittest.py there is a custom habit of simply replacing the "runner" while with py.test it's most often better to write just a custom reporter or a custom collector. Eventually, the py.test-config file should allow to have specific collectors, reporters and maybe even runners for subdirectories. Architecting such a federated scheme of collectors/runners/reporters is not easy but i think very much worth it. Note, that it should always be possible to run tests of any application with py-tests by invoking 'py.test APP-DIRECTORY' or simply 'py.test' while being in the app directory. At some point we may look into providing direct support for unittest.py style tests to allow a seemless "upgrade". But this may be extremely messy with all those unittest.py hacks around. > * I'm not clear how doctest would fit in. I guess I could turn the > doctest into a unit test TestCase, then test that. Of course, it would > be nice if this was streamlined. I also have fiddled with doctest to > use my own comparison functions when testing if we get the expected > output. That's not really in the scope of utest -- that should really > go in doctest. Anyway, I thought I'd note its existance. Yes, well noted. I think everyone agrees that integrating doctest is a good idea. I am just not using them currently but i like the idea. It's open how to integrate doctests into py.test. I guess the rough idea is to to extend the current collectors to look for docstrings and they would generate specific Units whose execute() method is invoked by our runner. The DoctestUnit.execute method would run a doctest. Probably, this also requires extending the TextReporter to support some nice kind of failure output. > * Code coverage tracking. This should be fairly straight-forward to add. yes. I guess everybody uses sys.settrace which is unfortunately rather expensive. Anyway, it would be nice to come up with some real life use cases and see what is really useful. I am not clear on that. > The last time I looked around at test runners, Zope3's seemed the best. > Well, it would have been better if I could have gotten it to do > something. But it *seemed* best. Mining it for features: > > * Different levels of tests (-a --at-level or --all; default level is 1, > which doesn't run all tests). They have lots of tests, so I'm guessing > they like to avoid running tests which are unlikely to fail. having different levels of tests seems interesting. I'd like more of a keyword based approach where all tests are associated with some keywords and you can select tests by providing keywords. Keywords could be automatically associated from filename and python name components e.g. ['std', 'path', 'local', 'test_listdir']. You could additionally associate a 'slow' keyword to some tests (somehow) and start 'py.test --exclude="slow"' or put this as a default in the configuration file. > * A distinction between unit and functional tests (as in acceptance or > system tests). This doesn't seem very generic -- these definitions are > very loose and not well agreed upon. There's not even any common > language for them. I'm not sure how this fits in with level, but some > sort of internal categorization of tests seems useful. maybe use the keyword approach for this, too? > * A whole build process. I think they run out of the build/ directory > that distutils generates. It is a little unclear how paths work out > with utest, depending on where you run it from. Setting PYTHONPATH to > include your development code seems the easiest way to resolve these > issues with utest. I don't have anything with complicated builds, so > maybe there's issues I'm unaware of. Simply providing a distutils-install should pose no problem and we should do it, however there are a couple of interesting issues here: - armin and me want a mechanism by which to include '.c' files in the library and seemlessly compiling (and possibly caching) them via distutils-mechanism. This should allow to work with a svn-checkout containing c-coded modules without any explicit user interaction and especially without distutils-installing it. - managing the py library versions in the long run, i'd like to have an easy automated way for users of the py lib to get/install a new version. preferably via a 'py' binary which also allows to ask 'py --version', 'py --selftest' to let the py-lib tests run, 'py --test' which would iterate into all modules/packages to find tests and run them. This is kind of integrity test for your system. Also 'py someprogram.py' could start an (interactive) python interpreter allowing the script to choose/restrict it's version (on all platforms). - eventually i'd like to think some more about the notion of a 'py' application which would be installable/manageable probably connecting to the PyPI project but including downloads. > * A pychecker option. (-c --pychecker) makes sense, although i am not using it. I have the suspicion that it would yell at py.magic :-) > * A pdb option (-D --debug). I was able to add this to utest with > fairly small modifications (at least, if I did it correctly). yes, nice, thanks. > * An option to control garbage collection (-g --gc-threshold). I guess > they encounter GC bugs sometimes. i'll let Armin comment on this :-) > * Run tests in a loop (-L --loop). Also for checking memory leaks. > I've thought that running loops of tests in separate threads could also > be a useful test, for code that actually was supposed to be used with > threads. That might be another place for specializing the runner. Yes, see above. Maybe 'py.test --session' could actually not return until all tests pass and wait for changing files to order to try again. This is really nice because you can just save from your editor and see the tests running (at least if you are working in multiple windows or on Xinerama like i do :-) > * Keep bytecode (-k --keepbytocode). Not interesting in itself, but it > implies that they don't normally keep bytecode. I expect this is to > deal with code where the .py file has been deleted, but the .pyc file is > still around. I've wasted time because of that before, so I can imagine > its usefulness. yes, there are some weird problems with py.test sometimes i haven't really looked into yet. Dealing with the compiled code will even get more of an issue when we let the tests run via "execnet" (see my other post). In this case the runner might invoke multiple python interpreters and once and run tests in parallel and it would be good to not simply write .pyc files at the same places where the .py files live. Wasn't there some option to python proposed to explicitely control the location where .pyc files are created? > * Profiling (-P --profile). Displays top 50 items, by time and # of calls. yip. Especially since the hotshot API is slightly verbose to use. > * Report only first doctest failure (-1 > --report-only-first-doctest-failure). yip. > * Time the tests and show the slowest 50 tests (-t --top-fifty). I > first thought this was just a bad way of doing profiling, but now that I > think about it this is to diagnose problems with the tests runnning slowly. Yes! > That's all the interesting options, I think. There's also options to > select which tests you display, but these seem too complex, while still > not all that powerful. See my idea about federated collectors / reporters / runners. If we get this right then such interesting options become very viable. OK, enough for now, i guess. Ian, thanks for coming to us and helping to move the py lib along! While i currently am the main driver i am very happy to share decisions, ideas and work, especially with knowledgable python developers. cheers, holger From hpk at trillke.net Fri Oct 1 19:06:07 2004 From: hpk at trillke.net (holger krekel) Date: Fri, 1 Oct 2004 19:06:07 +0200 Subject: [py-dev] background/future plans Message-ID: <20041001170607.GO7311@solar.trillke.net> Hi everbody, here is another mail to give and discuss some more background, part of which some people already know about, other's don't. First of all, there is a pending renaming from 'std' to 'py' accompanied by some subsequent small renamings. The rationale is - speaking of the 'std' lib is confusing for python-dev and other pythonistas - the name 'py' is pretty indicative and 'py' does support - like stated at EuroPython 2004 - the notion of beeing a "complementary second standard library". - command line tools provided by 'py' will have a prefix of 'py.*' like e.g. 'py.test' so that -completion works nicely with 'py.' both inside bash/readline as well as on the interactive python prompt. (i plan to enhance and integrate rlcompleter2 at some point, see http://codespeak.net/rlcompleter2) I'd like to do the renaming this weekend but am not sure yet how to do it. "Bicycle Repair Man" seems to be completly lacking any documentation and i know of know of no other tool to support such a grand scale renaming except writing my own little script. Also 'utest' is probably to become simply 'py.test' being a python namespace and the name of the central script. std-soon-to-be-py uses a strict mechanism to control the export of its names. In std/__init__.py you'll currently find all the namespaces and names listed which are exposed via the import mechanism. This allows the library to be easily refactorable even across major releases while retaining automatically tested backward compatibility. The idea is that - unless something gets deprecated - such "import APIs" are guaranteed across major releases. If it works in X.Y it will work in X.(Y+1). If it is deprecated in X.Y then it may go away in X.(Y+1). The behaviour and the names _indirectly_ reachable through classes are only guaranteed through the tests. So if you wan't to make sure that a certain behaviour lasts across releases you'd better write a test for it and check it in/submit it. Any untested API part could go away. OK, then on with some short introduction to the current py components - already using the proposed naming scheme: py the 'py lib hook' and root namespace. It's always enough to just "import py" and work your way with e.g. 'py.path.svnurl(...)'. Packages and modules will be lazily imported. All namespaces on the way should be clean and should have no garbage. (do a .path.local the local filesystem class providing an object oriented OO API for filesystem accesses .path.svnurl the remote subversion URL class providing listdir(), update(), remove(), ... methods for accesses to the transactional subversion filesystem .path.svnwc a local working copy API offering most of what path.local offers plus commit(), revert(), add() etc.pp. .test.main() the commandline API for starting tests .test.assert_raises the one method besides the plain 'assert' statement that you usually want from a testing module .test.compat.* some unittest.py compatibility stuff like TestCase .test.* various stuff that shouldn't maybe be exposed :-) .magic.dyncode a module which helps with dynamic code generation allowing for better tracebacks and some more convenience .magic.invoke/revoke the mechanism to invoke magic which usually involves (careful) monkey-patching of system libraries like inspect or the builtin namespace .execnet offering an innovative approach to networked applications by allowing them to execute code in each other's process space. Currently, this works for sockets and pipes/child processes, offering a 'SSHGateway' should be no problem as it is basically the same as the "PopenGateway". The small code in 'execnet' is an extract from a hacking week with Armin on a multiuser multiplatform multi-line concurrent pygame-based interactive python shell named shpy. (also in the codespeak repo). As you probably have noticed there are all kinds of little open ends and food for thought. More than anything, 'the py lib' should allow for a much faster pace and refactoring than the current python lib. Thus I find it most important to agree on mechanisms for making this possible mainly "export" control and tests-are-the-only-behaviour-guaranties at the moment. Again, the testing mechanism plays a central role. cheers, holger From ianb at colorstudy.com Fri Oct 1 20:05:32 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 01 Oct 2004 13:05:32 -0500 Subject: [py-dev] utest thoughts In-Reply-To: <20041001164412.GN7311@solar.trillke.net> References: <4158A934.4040800@colorstudy.com> <20041001164412.GN7311@solar.trillke.net> Message-ID: <415D9C6C.5000702@colorstudy.com> I thought I'd split this up, but most of it comes down to the same subject -- how to find tests, how to annotate tests, how to select tests, and those are all kind of the same problem. Well, that and some more minor details... holger krekel wrote: > Yes, selecting test methods by (maybe wildcarded) expressions seems > reasonable. However, i would like to first implement an > "stateful testing" approach which would basically work like this: > > py.test --session > ... runs all tests, some of which fail ... > > py.test --session > ... runs only the previously failed tests ... > > repeat the last step until all tests pass, > then run all tests --> go to step 1 > > I somehow think that this might satisfy many use cases which otherwise > would require some wildcard based selection scheme. That would definitely be a nice feature. There are still some use cases for wildcards. One would be when you are in a large package, and make a localized change, and you don't want to spend the time to run all the tests before you get to the code you just changed; maybe you'd run all the tests later, but you want to start with specific tests and then expand once those pass. Another is TDD; in that case you may be writing many tests that are expected to fail. You probably want to address the tests in a specific order, and not filter through all the other tests at the same time. > py.test consists of three main elements when "running" tests: > the collector(s), the reporter (for producing all output), > and the runner, driving the collector and the reporter. Collectors > are completly decoupled from running tests. They are implemented > (std/utest/collect.py) via a concept called "Restartable Iterators", > thus the runner doesn't not need to know all the tests when starting > to run them. > > I know that unittest.py approaches often just put all these > three elements together and call it a "runner" but i think > it blurs the lines and makes it very difficult to keep it > extensible and customizable. > > Having said this, for having multiple data sets it's probably best > to introduce a mechanism into the Module Collector to look for > custom module-defined collectors which would seamlessly integrate > into the collection process of Units. By implementing your own > collection function you can yield many Units with different > test data sets which are then executed/accounted for individually. Could this be like: data = [(1, 'one'), (2, 'two'), ...] def test_collector(): for args in data: # Depending on when tests are run, I think leaving out args=args # could silently make all your tests use the same data :( yield lambda args=args: test_converter(*args) # Should we then to test_collector = test_collector() ? # Maybe test_converter is a bad name, because the runner will find it... # or if test_collector is present, maybe the runner won't look any # further in this module; but sometimes we will want it to look through # the module... def test_converter(number, english): assert make_english(number) == english That would be pleasingly simple. I'm hoping we can avoid exposing complex interfaces to the test code. It gets more complicated if you can address the tests individually, e.g., via wildcards or keywords. How to add that information to the individual tests? >>* Specifying an option to the runner that gets passed through to the >>tests. It seems like the options are fixed now. I'd like to do >>something like -Ddatabase=mysql. I can do this with environmental >>variables now, but that's a little crude. It's easiest if it's just >>generic, like -D for compilers, but of course it would be nicer if there >>were specific options. Maybe this could be best achieved by >>specializing utest and distributing my own runner with the project. > > > I see the point. Doing this should be done by reexaminig the current > 'config' mechanism. Or better: rewriting it alltogether as it is very > much ad hoc/preliminary. There is a provision to have utest configuration > files currently called utest.conf where you can at the moment only > set some default values for command line options. > > with unittest.py there is a custom habit of simply replacing > the "runner" while with py.test it's most often better to write > just a custom reporter or a custom collector. > > Eventually, the py.test-config file should allow to have specific > collectors, reporters and maybe even runners for subdirectories. Hmm... this would address certain issues. For instance, if you're doing functional tests on a web app, you might configure what URL the app is locally installed at. In the case I'm thinking of, where I run the identical set of tests on different backends, it should be available as a command-line argument. But if there's a generic command-line argument (like -D) then that could be used to set arbitrary options (assuming the config file can accept arbitrary options). > Architecting such a federated scheme of collectors/runners/reporters > is not easy but i think very much worth it. > > Note, that it should always be possible to run tests of any > application with py-tests by invoking 'py.test APP-DIRECTORY' or > simply 'py.test' while being in the app directory. What about "python setup.py test" ? This would allow for a common way to invoke tests, regardless of runner; people could use this for their unittest-based tests as well, or whatever they are using. I think I tried this at one point, but then got bored of trying to figure out the distutils just to add this one little command. > At some point we > may look into providing direct support for unittest.py style tests > to allow a seemless "upgrade". But this may be extremely messy > with all those unittest.py hacks around. > >>* I'm not clear how doctest would fit in. I guess I could turn the >>doctest into a unit test TestCase, then test that. Of course, it would >>be nice if this was streamlined. I also have fiddled with doctest to >>use my own comparison functions when testing if we get the expected >>output. That's not really in the scope of utest -- that should really >>go in doctest. Anyway, I thought I'd note its existance. > > > Yes, well noted. I think everyone agrees that integrating doctest > is a good idea. I am just not using them currently but i like the > idea. It's open how to integrate doctests into py.test. I guess > the rough idea is to to extend the current collectors to look for > docstrings and they would generate specific Units whose execute() > method is invoked by our runner. The DoctestUnit.execute method > would run a doctest. Probably, this also requires extending > the TextReporter to support some nice kind of failure output. In Zope3 they explicitly add doctests to the testing, it isn't just automatically picked up. In part because the doctests are typically in modules that aren't otherwise inspected for tests (they are inline with the normal code, not in seperate test_* modules). I think there may be a performance issue with inspecting all modules for doctests, and potentially an issue of finding things that look like tests but aren't (though that probably isn't a big problem, since there's ways to exclude docstrings from doctest). >>* Code coverage tracking. This should be fairly straight-forward to add. > > > yes. I guess everybody uses sys.settrace which is > unfortunately rather expensive. Anyway, it would be nice to > come up with some real life use cases and see what is really > useful. I am not clear on that. I think the "50% code coverage" is mostly a feel-good measure, so you can be pleased with your increasing score as you add tests. It would be awesome to allow for leveling up with your tests. "20% code coverage; you have begun your travels" or "95% code coverage; you are approaching enlightenment". The actual file-by-file reports are more useful, I think. Coverage should only be tracked when explicitly asked for. >>* Different levels of tests (-a --at-level or --all; default level is 1, >>which doesn't run all tests). They have lots of tests, so I'm guessing >>they like to avoid running tests which are unlikely to fail. > > > having different levels of tests seems interesting. I'd like > more of a keyword based approach where all tests are > associated with some keywords and you can select tests by > providing keywords. Keywords could be automatically > associated from filename and python name components e.g. > ['std', 'path', 'local', 'test_listdir']. You could > additionally associate a 'slow' keyword to some tests (somehow) > and start 'py.test --exclude="slow"' or put this as a default in > the configuration file. That would work well, I think. How might tests be annotated? On a module-by-module basis, using some particular symbol (__test_keywords__)? Function attributes? On an ad hoc basis by customizing the collector? It's actually the kind of place where adaptation would be interesting; objects would be adapted to test cases, where part of the test case API was a set of keywords. That would allow for a lot of customization, while the actual tests could remain fairly simple. Part of the base of py.test would be adapters for packages, modules, and functions; the module adapter looks for the test_* functions, function adapters might look for function attributes, etc. There'd be another adapter for unittest.TestCase and unittest.TestSuite, and so on. Packages could create their own adapters for further customization. >>* A distinction between unit and functional tests (as in acceptance or >>system tests). This doesn't seem very generic -- these definitions are >>very loose and not well agreed upon. There's not even any common >>language for them. I'm not sure how this fits in with level, but some >>sort of internal categorization of tests seems useful. > > > maybe use the keyword approach for this, too? Seems like a good use case. >>* A whole build process. I think they run out of the build/ directory >>that distutils generates. It is a little unclear how paths work out >>with utest, depending on where you run it from. Setting PYTHONPATH to >>include your development code seems the easiest way to resolve these >>issues with utest. I don't have anything with complicated builds, so >>maybe there's issues I'm unaware of. > > > Simply providing a distutils-install should pose no problem and > we should do it, however there are a couple of interesting issues here: > > - armin and me want a mechanism by which to include '.c' > files in the library and seemlessly compiling (and > possibly caching) them via distutils-mechanism. This > should allow to work with a svn-checkout containing > c-coded modules without any explicit user interaction > and especially without distutils-installing it. Right, this is what Zope is doing. It builds the package (but does not install it) before running the tests (python setup.py build). Then it runs the tests out of the build/ directory. The build is, I think, relatively fast (after the first time it is run); it only updates things according to timestamps. > - managing the py library versions in the long run, i'd > like to have an easy automated way for users of the py lib > to get/install a new version. preferably via a 'py' binary > which also allows to ask 'py --version', 'py --selftest' > to let the py-lib tests run, 'py --test' which would iterate > into all modules/packages to find tests and run them. This is > kind of integrity test for your system. Also 'py someprogram.py' > could start an (interactive) python interpreter allowing > the script to choose/restrict it's version (on all platforms). > > - eventually i'd like to think some more about the notion > of a 'py' application which would be installable/manageable > probably connecting to the PyPI project but including > downloads. Is there a reason these separate concerns go together? The last seems like a distutils enhancement. Handling multiple versions... well, that's another issue that is pretty much unaddressed at this point, but I'm not sure > >>* Run tests in a loop (-L --loop). Also for checking memory leaks. >>I've thought that running loops of tests in separate threads could also >>be a useful test, for code that actually was supposed to be used with >>threads. That might be another place for specializing the runner. > > > Yes, see above. Maybe 'py.test --session' could actually not return > until all tests pass and wait for changing files to order to try again. > This is really nice because you can just save from your editor and > see the tests running (at least if you are working in multiple windows > or on Xinerama like i do :-) I think for a text reporter this would lead to information overload. In Zope I assume they'd only use this once all tests passed, as a way of exercising the C code. >>* Keep bytecode (-k --keepbytocode). Not interesting in itself, but it >>implies that they don't normally keep bytecode. I expect this is to >>deal with code where the .py file has been deleted, but the .pyc file is >>still around. I've wasted time because of that before, so I can imagine >>its usefulness. > > > yes, there are some weird problems with py.test sometimes i haven't really > looked into yet. Dealing with the compiled code will even get more of an > issue when we let the tests run via "execnet" (see my other post). In this > case the runner might invoke multiple python interpreters and once and > run tests in parallel and it would be good to not simply write .pyc > files at the same places where the .py files live. Wasn't there some > option to python proposed to explicitely control the location where > .pyc files are created? There was a PEP, and some (mostly positive) discussion, but it lost momentum and got lost. PEP 304, I think: http://www.python.org/peps/pep-0304.html -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Oct 1 20:09:51 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 01 Oct 2004 13:09:51 -0500 Subject: [py-dev] background/future plans In-Reply-To: <20041001170607.GO7311@solar.trillke.net> References: <20041001170607.GO7311@solar.trillke.net> Message-ID: <415D9D6F.4010807@colorstudy.com> holger krekel wrote: > .path.local the local filesystem class providing an > object oriented OO API for filesystem accesses > > .path.svnurl the remote subversion URL class providing > listdir(), update(), remove(), ... > methods for accesses to the transactional > subversion filesystem > > .path.svnwc a local working copy API offering most of what > path.local offers plus commit(), revert(), add() > etc.pp. Do you have thoughts on the path module (http://www.jorendorff.com/articles/python/path/). I've used it before some (mostly I forget to use it), and it's pretty complete. My only criticism might be that there's a few too many methods to keep track of. Anyway, it's close but not identical to path.local, and it's been used and had feedback from a fair number of people. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From hpk at trillke.net Fri Oct 1 21:01:40 2004 From: hpk at trillke.net (holger krekel) Date: Fri, 1 Oct 2004 21:01:40 +0200 Subject: [py-dev] background/future plans In-Reply-To: <415D9D6F.4010807@colorstudy.com> References: <20041001170607.GO7311@solar.trillke.net> <415D9D6F.4010807@colorstudy.com> Message-ID: <20041001190140.GQ7311@solar.trillke.net> [Ian Bicking Fri, Oct 01, 2004 at 01:09:51PM -0500] > holger krekel wrote: > > .path.local the local filesystem class providing an > > object oriented OO API for filesystem accesses > > > > .path.svnurl the remote subversion URL class providing > > listdir(), update(), remove(), ... > > methods for accesses to the transactional > > subversion filesystem > > > > .path.svnwc a local working copy API offering most of what > > path.local offers plus commit(), revert(), add() > > etc.pp. > > Do you have thoughts on the path module > (http://www.jorendorff.com/articles/python/path/). I've used it before > some (mostly I forget to use it), and it's pretty complete. My only > criticism might be that there's a few too many methods to keep track of. > Anyway, it's close but not identical to path.local, and it's been used > and had feedback from a fair number of people. while path.local is only used by a few projects it is well tested and shares many methods with the other path.svnurl and path.svnwc implementations. I don't think there are e.g. subversion or other implementations of the ohrendorff-interface. Also, i think it's a questionable idea to subclass Path objects from string or unicode which introduces lots of names, some of which are superflous or even confusing in a filesystem context. The visit/listdir methods of py.path.* objects are rather advanced, e.g. for x in pathobj.visit(fil=py.path.checker(file=1, link=0, fnmatch='*.py'), rec=py.path.checker(dotfile=0)): # x will iterate over all .py files which are not # links and will not recurse into "." directories and this, of course, works on all path implementations. And, of course, you can pass full-blown functions as filter and recursion parameters. py.path.checker is just there to cover some common ground. Another feature, especially nice with py.path.local objects, is that exceptions are transformed so you'll write e.g.: try: localpathobj.somemethod() except py.path.NotFound: ... except py.path.Denied: ... except py.path.IsDirectory: ... to handle certain problems while accessing filesystems. Note that this is _not_ generic yet, i.e. it doesn't work the same way with subversion paths. Other than that, some things are similar, e.g. recently py.path.* objects grew basename, ext and purebasename read-only properties. In general, i believe that interacting with remote or local Filesystem Objects is something which a higher-level library should support very well and consistently. The Ohrendorff seems to only go part of the way. cheers, holger From hpk at trillke.net Sun Oct 3 09:41:30 2004 From: hpk at trillke.net (holger krekel) Date: Sun, 3 Oct 2004 09:41:30 +0200 Subject: [py-dev] std rename to py done Message-ID: <20041003074130.GW7311@solar.trillke.net> hello, i finished most of the rename from 'std' to 'py' yesterday night. At least "py.test" (formerly utest) completes the tree without errors. the new "trunk" of the py lib is at http://codespeak.net/svn/py/dist so i left out 'trunk' because i think it makes the url/svn namespace unneccesarily nested. The above URL currently carries the 'py' package along with an (alas, empty :-) doc directory. Instead of writing long emails i intend to write ReST docs. IMPORTANT, please stop committing anything to 'std' which i will set to readonly in short order, i guess. If you _have_ any changes pending then commit them now, i should be able to automatically port these changes to the py lib. Btw, in case you are interested, I have written my own hack (using the parser and tokenizer) for doing the completely automatic refactoring/rename/moves in http://codespeak.net/svn/user/hpk/rename_import.py which expects the target argument to be the resulting directory of a svn cp path/to/your/WC/of/std path/to/new/py operation. You can repeat the rename_import.py as it does a "revert -R" before performing the rename/refactor. This allows for svn diffing the results of the rename. This rename_import.py script can also be 'utest'ed as it contains inline tests. Obviously, rename_import.py requires 'std' not 'py' for apparent reasons but i plan to port and improve the script, and make it available as an example script (it uses the local file and svn working copy path objects as well as in-lined tests). cheers, holger From hpk at trillke.net Sun Oct 3 12:46:19 2004 From: hpk at trillke.net (holger krekel) Date: Sun, 3 Oct 2004 12:46:19 +0200 Subject: [py-dev] py-svn mailing list Message-ID: <20041003104619.GZ7311@solar.trillke.net> hi, i've just created a py-svn list which receives all updates to the http://codespeak.net/svn/py/dist tree. Please subscribe here: http://codespeak.net/mailman/listinfo/py-svn cheers, holger From hpk at trillke.net Sun Oct 3 13:40:16 2004 From: hpk at trillke.net (holger krekel) Date: Sun, 3 Oct 2004 13:40:16 +0200 Subject: [py-dev] utest thoughts In-Reply-To: <415D9C6C.5000702@colorstudy.com> References: <4158A934.4040800@colorstudy.com> <20041001164412.GN7311@solar.trillke.net> <415D9C6C.5000702@colorstudy.com> Message-ID: <20041003114016.GA7311@solar.trillke.net> Hi Ian, hi all, [Ian Bicking] > holger krekel wrote: > >Having said this, for having multiple data sets it's probably best > >to introduce a mechanism into the Module Collector to look for > >custom module-defined collectors which would seamlessly integrate > >into the collection process of Units. By implementing your own > >collection function you can yield many Units with different > >test data sets which are then executed/accounted for individually. > > Could this be like: > > data = [(1, 'one'), (2, 'two'), ...] > def test_collector(): > for args in data: > # Depending on when tests are run, I think leaving out args=args > # could silently make all your tests use the same data :( > yield lambda args=args: test_converter(*args) > # Should we then to test_collector = test_collector() ? yes, though a) the collection function should not have a 'test' pre- or postfix i think. b) collectors always yield collectors or Units, not plain functions so you would actually do yield py.test.Unit(my_custom_test_function, *args) In the current py code (see my other postings) you'll find in py/test/test/data/Collector.py the current way to do custom collectors. This Collector class is instantiated with the module's location and takes over the collecting process for the module. No further attempt is made to collect anything from a module which contains a custom "Collector" object. > In the case I'm thinking of, where I run the identical set of tests on > different backends, it should be available as a command-line argument. > But if there's a generic command-line argument (like -D) then that could > be used to set arbitrary options (assuming the config file can accept > arbitrary options). you already took '-D' for the pdb debugging but I guess we don't need "--pdb" as a short option. I took the freedom to rename it from "--usedpdb" by the way. Introducing "-D" for passing options/backends to the tests still requires more careful thoughts about test configuration (files) in general. I think it should be possible to have test-configuration files per directory, which might modify the collection process and deal with configuration data ("-D") issues. They _may_ also provide a different runner if it turns out to make sense. Again please note, that py.test's "runner" has simpler responsibilities than unittest.py - based runners. We have the runner, the collectors and the reporter. > >Note, that it should always be possible to run tests of any > >application with py-tests by invoking 'py.test APP-DIRECTORY' or > >simply 'py.test' while being in the app directory. > > What about "python setup.py test" ? This would allow for a common way > to invoke tests, regardless of runner; people could use this for their > unittest-based tests as well, or whatever they are using. I think it's easier to say py.test APP-DIRECTORY and i don't want to deal with distutils-hacks if it can be avoided ... > I think I tried this at one point, but then got bored of trying to > figure out the distutils just to add this one little command. ... we seem to share the same view here :-) > In Zope3 they explicitly add doctests to the testing, it isn't just > automatically picked up. Argh! I have to say i dislike writing repetitive unneccessary "manually-synced" boilerplate code for tests, be they unit, doc or any other kind of test. > In part because the doctests are typically in modules that > aren't otherwise inspected for tests (they are inline with > the normal code, not in seperate test_* modules). I think > there may be a performance issue with inspecting all modules > for doctests, and potentially an issue of finding things > that look like tests but aren't (though that probably isn't > a big problem, since there's ways to exclude docstrings from > doctest). Hum, performance problem. I think importing all test and implementation code of the py lib requires 2 seconds on a file system. As the collection happens iterative these two seconds are distributed across the whole testing time. With "Test-Session" modes it will be less. But if there is a performance problem we may think of ways to exclude files from looking for doctests by means of the per-directory py.test file. > >>* Code coverage tracking. This should be fairly straight-forward to add. > > > >yes. I guess everybody uses sys.settrace which is > >unfortunately rather expensive. Anyway, it would be nice to > >come up with some real life use cases and see what is really > >useful. I am not clear on that. > > I think the "50% code coverage" is mostly a feel-good measure, so you > can be pleased with your increasing score as you add tests. It would be > awesome to allow for leveling up with your tests. "20% code coverage; > you have begun your travels" or "95% code coverage; you are approaching > enlightenment". yes, sounds nice. > The actual file-by-file reports are more useful, I think. Coverage > should only be tracked when explicitly asked for. Yes, i guess so. The same is true for profiling. > >>* Different levels of tests (-a --at-level or --all; default level is 1, > >>which doesn't run all tests). They have lots of tests, so I'm guessing > >>they like to avoid running tests which are unlikely to fail. > > > >having different levels of tests seems interesting. I'd like > >more of a keyword based approach where all tests are > >associated with some keywords and you can select tests by > >providing keywords. Keywords could be automatically > >associated from filename and python name components e.g. > >['std', 'path', 'local', 'test_listdir']. You could > >additionally associate a 'slow' keyword to some tests (somehow) > >and start 'py.test --exclude="slow"' or put this as a default in > >the configuration file. > > That would work well, I think. How might tests be annotated? On a > module-by-module basis, using some particular symbol > (__test_keywords__)? Function attributes? On an ad hoc basis by > customizing the collector? It has to be possible to add keywords (i think we should only ever deal with adding keywords) on the module, class and method level. Finding syntax for the module level and the class level boils down to thinking about a good name which lists the additional keywords. For the method level we would have to worry about syntax so i'd like to avoid it alltogether. I think that with "automatic keywords", derived from the name of the test module, class (if any) and method name we are all set. You can always "add" a keyword by extending the testname, avoiding any redundancy. For example, we could introduce "bug" tests: def test_bug001_pypath_sucks(): # ... and then you could run a specific "bug001" test by something like py.test --include="bug001" note, btw, that i plan to distribute ".svn" directories along with every copy of 'py'. This allows _everyone_ commit access to test_*.py and *_test.py files in the repository! So if someone finds a problem or wants a guarantee from the py lib he can simply contribute it! Committing to the implementation tree will require registration for an account, though. > It's actually the kind of place where adaptation would be interesting; > objects would be adapted to test cases, where part of the test case API > was a set of keywords. That would allow for a lot of customization, > while the actual tests could remain fairly simple. Part of the base of > py.test would be adapters for packages, modules, and functions; the > module adapter looks for the test_* functions, function adapters might > look for function attributes, etc. There'd be another adapter for > unittest.TestCase and unittest.TestSuite, and so on. Packages could > create their own adapters for further customization. I am not sure, i am following. What would be the distinct advantates of introducing this machinery? > > - armin and me want a mechanism by which to include '.c' > > files in the library and seemlessly compiling (and > > possibly caching) them via distutils-mechanism. This > > should allow to work with a svn-checkout containing > > c-coded modules without any explicit user interaction > > and especially without distutils-installing it. > > Right, this is what Zope is doing. It builds the package (but does not > install it) before running the tests (python setup.py build). We mean it a lot more automatic. You should not need to run _any_ intermediate command, no matter how simple. A checkout of the py lib should be enough. And you can simply modify your '.c' file somewhere under the implementation tree and nicely expose any objects implemented via the "package export" runtime mechanism. The motto of the py lib is "avoiding APIs" :-) > > - managing the py library versions in the long run, i'd > > like to have an easy automated way for users of the py lib > > to get/install a new version. preferably via a 'py' binary > > which also allows to ask 'py --version', 'py --selftest' > > to let the py-lib tests run, 'py --test' which would iterate > > into all modules/packages to find tests and run them. This is > > kind of integrity test for your system. Also 'py someprogram.py' > > could start an (interactive) python interpreter allowing > > the script to choose/restrict it's version (on all platforms). > > > > - eventually i'd like to think some more about the notion > > of a 'py' application which would be installable/manageable > > probably connecting to the PyPI project but including > > downloads. > > Is there a reason these separate concerns go together? The last seems > like a distutils enhancement. Handling multiple versions... well, > that's another issue that is pretty much unaddressed at this point, but > I'm not sure Well, we need not discuss this further right now, i guess. I just wanted to present some of the ideas i am having. I do think that the notion of a "py app" which works well with the py-commandline utilities, is installable and upgradeable seemlessly from remote places is something to go for. It would be more than distutils and probably sitting on top of it and PyPI. > >>* Run tests in a loop (-L --loop). Also for checking memory leaks. > >>I've thought that running loops of tests in separate threads could also > >>be a useful test, for code that actually was supposed to be used with > >>threads. That might be another place for specializing the runner. > > > >Yes, see above. Maybe 'py.test --session' could actually not return > >until all tests pass and wait for changing files to order to try again. > >This is really nice because you can just save from your editor and > >see the tests running (at least if you are working in multiple windows > >or on Xinerama like i do :-) > > I think for a text reporter this would lead to information overload. Well, the tests would only run (and fail) when implementation files got modified. So most of the time it would be sitting idly and print nothing. I have used this scheme, it's extremely nice, and doesn't overload with information. It just avoids typing the same commands over and over. cheers, holger From ianb at colorstudy.com Sun Oct 3 21:07:17 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 03 Oct 2004 14:07:17 -0500 Subject: [py-dev] utest thoughts In-Reply-To: <20041003114016.GA7311@solar.trillke.net> References: <4158A934.4040800@colorstudy.com> <20041001164412.GN7311@solar.trillke.net> <415D9C6C.5000702@colorstudy.com> <20041003114016.GA7311@solar.trillke.net> Message-ID: <41604DE5.8080703@colorstudy.com> holger krekel wrote: >>Could this be like: >> >>data = [(1, 'one'), (2, 'two'), ...] >>def test_collector(): >> for args in data: >> # Depending on when tests are run, I think leaving out args=args >> # could silently make all your tests use the same data :( >> yield lambda args=args: test_converter(*args) >># Should we then to test_collector = test_collector() ? > > > yes, though > > a) the collection function should not have a 'test' pre- or postfix > i think. Sure. > b) collectors always yield collectors or Units, not plain functions > so you would actually do > > yield py.test.Unit(my_custom_test_function, *args) > > In the current py code (see my other postings) you'll find > in py/test/test/data/Collector.py the current way to do custom > collectors. This Collector class is instantiated with > the module's location and takes over the collecting > process for the module. No further attempt is made > to collect anything from a module which contains a > custom "Collector" object. I have a problem with APIs that jump in complexity. In this case you can forget about Unit until you start needing to generate multiple tests programmatically -- your functions *were* your units, until that point. If functions are units in one place, they should be units everyplace. This is where adaptation is a potentially useful idea. Instead of asking for a specific interface for any particular object, you are only asking for an object that can be adapted to a particular interface. You do this adaptation at the last possible moment, so that something like a custom collector doesn't produce Units, but merely produces something that can be turned into a Unit (or rather, turned into something that supports the IUnit interface). Doing it this way, functions really *are* units, for all purposes. Also, you've introduced a decoupled abstraction. The tests can be any kind of object, whatever makes sense for the project -- maybe functions, maybe doctests, maybe classes, and so on. The runner iterates over unites. The collector is an adapter from packages to units, and recursively it adapts from modules to units, from functions to units, etc. But with adaptation the collector isn't something that the runner starts up, rather it is a registered service. So customization might simply mean that the runner starts by loading __init__, or test_init, or something like that. That module can register adapters which are more specialized than the standard adapters, or override the standard adapters for a subset of the package. (How to get them to apply to a subset? I'm not sure, that's not a metaphor I've seen with adaptation, though maybe multiple-value adapters apply somehow -- I'm still new at using adaptation.) >>In the case I'm thinking of, where I run the identical set of tests on >>different backends, it should be available as a command-line argument. >>But if there's a generic command-line argument (like -D) then that could >>be used to set arbitrary options (assuming the config file can accept >>arbitrary options). > > > you already took '-D' for the pdb debugging but I guess we don't need > "--pdb" as a short option. I took the freedom to rename it from > "--usedpdb" by the way. Sure. I just took the option name from Zope, but I don't really care what it is. > Introducing "-D" for passing options/backends to the tests > still requires more careful thoughts about test configuration > (files) in general. I think it should be possible to have > test-configuration files per directory, which might modify > the collection process and deal with configuration data ("-D") > issues. They _may_ also provide a different runner if it > turns out to make sense. Again please note, that py.test's > "runner" has simpler responsibilities than unittest.py - based > runners. We have the runner, the collectors and the reporter. I know, I think I say "runner" to mean "the whole process". If we are assuming that the tests are run in serial, configuration could be done through an initialization hook. So you just put something in like: # in test_init.py... import py.test def test_setup(): py.test.options.oldusepdb = py.test.options.usepdb py.test.options.usepdb = True def test_teardown(): py.test.options.usepdb = py.test.options.oldusepdb del py.test.options.oldusepdb Kind of lame, but maybe functional. >>>Note, that it should always be possible to run tests of any >>>application with py-tests by invoking 'py.test APP-DIRECTORY' or >>>simply 'py.test' while being in the app directory. >> >>What about "python setup.py test" ? This would allow for a common way >>to invoke tests, regardless of runner; people could use this for their >>unittest-based tests as well, or whatever they are using. > > > I think it's easier to say > > py.test APP-DIRECTORY > > and i don't want to deal with distutils-hacks if it can be avoided ... My only concern is that py.test not be too novel; making test creation accessible is a big motivation for me. I like your idea of encouraging bug reports to be done by committing a failing test. Acknowledging that other people are going to be running their tests using unittest, or unittest with custom runners, using setup.py is a way to provide a common entry point across all Python projects. >>I think I tried this at one point, but then got bored of trying to >>figure out the distutils just to add this one little command. > > > ... we seem to share the same view here :-) > > >>In Zope3 they explicitly add doctests to the testing, it isn't just >>automatically picked up. > > > Argh! I have to say i dislike writing repetitive unneccessary > "manually-synced" boilerplate code for tests, be they unit, > doc or any other kind of test. Yeah... but it's not *that* bad. It would probably look like: from py.test import make_doctest # <-- not a great name import somemodule doctest_collector = make_doctest(somemodule) # or... doctest_collector = [make_doctest(somemodule.SomeClass), make_doctest(somemodule.SomeClass2)] Maybe it's not that big a deal, because practically every module is going to be imported anyway. I don't know how much time it takes to parse docstrings; heck, you could cache that parsing too, like .pyc caches the compilation. Or you could look for the __test__ variable that doctest uses; then you'd have to use __test__ = {} when you didn't have any additional tests, but that's relatively little boilerplate... though that's probably not a good compromise. >>In part because the doctests are typically in modules that >>aren't otherwise inspected for tests (they are inline with >>the normal code, not in seperate test_* modules). I think >>there may be a performance issue with inspecting all modules >>for doctests, and potentially an issue of finding things >>that look like tests but aren't (though that probably isn't >>a big problem, since there's ways to exclude docstrings from >>doctest). > > > Hum, performance problem. I think importing all test and implementation > code of the py lib requires 2 seconds on a file system. As > the collection happens iterative these two seconds are distributed > across the whole testing time. With "Test-Session" modes it > will be less. > > But if there is a performance problem we may think of ways > to exclude files from looking for doctests by means of the > per-directory py.test file. > > >>>>* Different levels of tests (-a --at-level or --all; default level is 1, >>>>which doesn't run all tests). They have lots of tests, so I'm guessing >>>>they like to avoid running tests which are unlikely to fail. >>> >>>having different levels of tests seems interesting. I'd like >>>more of a keyword based approach where all tests are >>>associated with some keywords and you can select tests by >>>providing keywords. Keywords could be automatically >>>associated from filename and python name components e.g. >>>['std', 'path', 'local', 'test_listdir']. You could >>>additionally associate a 'slow' keyword to some tests (somehow) >>>and start 'py.test --exclude="slow"' or put this as a default in >>>the configuration file. >> >>That would work well, I think. How might tests be annotated? On a >>module-by-module basis, using some particular symbol >>(__test_keywords__)? Function attributes? On an ad hoc basis by >>customizing the collector? > > > It has to be possible to add keywords (i think we should > only ever deal with adding keywords) on the module, class and > method level. Finding syntax for the module level and the > class level boils down to thinking about a good name which > lists the additional keywords. > > For the method level we would have to worry about syntax > so i'd like to avoid it alltogether. Why not just attributes for all of these, e.g.: # module __test_keywords__ = ['bug001', 'pkg1'] def test_speed(): ... test_speed.__test_keywords__ = ['profile'] class SpeedTester: __test_keywords__ = ['profile'] def test_remote(self): ... test_remote.__test_keywords__ = ['urllib'] And so on. Since attributes can be added to anything, it provides a pretty easy interface. > I think that with "automatic keywords", derived from > the name of the test module, class (if any) and method > name we are all set. You can always "add" a keyword by > extending the testname, avoiding any redundancy. > > For example, we could introduce "bug" tests: > > def test_bug001_pypath_sucks(): > # ... > > and then you could run a specific "bug001" test by something like > > py.test --include="bug001" In some ways this seems simple, but I'd expect the keywords to be fairly volatile. I'm not sure that I like that the function names would have to be as volatile as the keywords; e.g., if you add a new keyword, will you rename half your functions? And the names also have to have other parts to describe them that aren't intended to be keywords, and these could be mistaken as keywords by the collector. > note, btw, that i plan to distribute ".svn" directories along with > every copy of 'py'. This allows _everyone_ commit access > to test_*.py and *_test.py files in the repository! So if > someone finds a problem or wants a guarantee from the py lib > he can simply contribute it! Committing to the implementation > tree will require registration for an account, though. That would be pretty neat. I wonder, could you allow anonymous commit access to a subset of the tree? I guess that's not too hard with Apache. I'd rather not let anyone willy-nilly commit test cases to the core; many perceived bugs aren't real bugs, and the tests might not fit project standards. I think a bugreport/ directory, where each bug is reported as a module containing a test case and perhaps a docstring explaining expected behavior, would be perfect. They can be moved into the main tests if appropriate, or just hang out there until they are reviewed, and you would allow anonymous commits in that directory. >>It's actually the kind of place where adaptation would be interesting; >>objects would be adapted to test cases, where part of the test case API >>was a set of keywords. That would allow for a lot of customization, >>while the actual tests could remain fairly simple. Part of the base of >>py.test would be adapters for packages, modules, and functions; the >>module adapter looks for the test_* functions, function adapters might >>look for function attributes, etc. There'd be another adapter for >>unittest.TestCase and unittest.TestSuite, and so on. Packages could >>create their own adapters for further customization. > > > I am not sure, i am following. What would be the distinct advantates > of introducing this machinery? I wrote about it more above, but the advantages would be a unified and reasonably transparent way of converting ad hoc or package-specific test cases into py.test test cases, and one that was decoupled from the tests themselves. E.g., supporting unittest.TestCase would just be a matter of providing an adapter. >>> - armin and me want a mechanism by which to include '.c' >>> files in the library and seemlessly compiling (and >>> possibly caching) them via distutils-mechanism. This >>> should allow to work with a svn-checkout containing >>> c-coded modules without any explicit user interaction >>> and especially without distutils-installing it. >> >>Right, this is what Zope is doing. It builds the package (but does not >>install it) before running the tests (python setup.py build). > > > We mean it a lot more automatic. You should not need to run _any_ > intermediate command, no matter how simple. A checkout of the > py lib should be enough. And you can simply modify your '.c' > file somewhere under the implementation tree and nicely expose > any objects implemented via the "package export" runtime mechanism. It would be automatic -- Zope's test.py automatically runs "setup.py build" (or whatever the internal method call is to do that) before every test run, then adjusts the path accordingly so code is loaded out of build/. It should be configurable, since for pure-python packages it's not necessary or desired -- it's just a bunch of copying that doesn't serve much purpose. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Oct 14 19:57:25 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 14 Oct 2004 12:57:25 -0500 Subject: [py-dev] utest docs; questions Message-ID: <416EBE05.5080500@colorstudy.com> Reading through the utest docs, I have a few questions: """ Managing test state across test modules, classes and methods ------------------------------------------------------------ Often you want to create some test files, db-connections or other state in order to run tests in a certain environment. With ``py.test`` there are three scopes for which you can provide hooks to manage such state. These hooks will get called by the driver:: """ Where are these functions put? At the module level? If so, why do they receive a module argument? I guess I can imagine:: from common import setup_module So that the setup_module function can work on multiple modules. Is this the intention? Maybe the example below (corrected if necessary) would make this all more clear. """ Another ``PyCollector`` then recurses into the test files and collects functions and methods that have a leading ``test_``. By default, methods are only collected if their class starts with ``Test``. """ It's not clear here how classes and methods work. Is the class instantiated with no arguments, then all the test_* methods run? E.g.: # in test_math.py class TestMath: def test_plus(self): n = self.num for i in range(self.count-1): n = self.num + n assert n == self.num * self.count def setup_class(cls): self.num = 5 self.count = 3 setup_class = classmethod(setup_class) def setup_method(self, method): # Sigh, I can't think of a good example... pass What happens here? E.g.: import test_math # if setup_module existed, call test_math.setup_module(test_math) TestMatch.setup_class() instance = TestMath() instance.setup_method(instance.test_plus) instance.test_plus() # instance.teardown_method(instance.test_plus) # TestMatch.teardown_class() # teardown_module(test_math) Obviously this can be customized by using different Items, but I'm thinking of the default action. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From hpk at trillke.net Thu Oct 14 21:00:27 2004 From: hpk at trillke.net (holger krekel) Date: Thu, 14 Oct 2004 21:00:27 +0200 Subject: [py-dev] utest docs; questions In-Reply-To: <416EBE05.5080500@colorstudy.com> References: <416EBE05.5080500@colorstudy.com> Message-ID: <20041014190027.GY15456@solar.trillke.net> Hi Ian, [Ian Bicking Thu, Oct 14, 2004 at 12:57:25PM -0500] > Reading through the utest docs, I have a few questions: > > """ > Managing test state across test modules, classes and methods > ------------------------------------------------------------ > > Often you want to create some test files, db-connections or > other state in order to run tests in a certain environment. > With ``py.test`` there are three scopes for which you can > provide hooks to manage such state. These hooks will get > called by the driver:: > """ > > Where are these functions put? At the module level? If so, why > do they receive a module argument? I guess I can imagine:: > > from common import setup_module exactly. > So that the setup_module function can work on multiple modules. Is this > the intention? Maybe the example below (corrected if necessary) would > make this all more clear. > yes, makes sense ... i just committed updated documentation and a working example (there is a new example-directory now) inspired by your suggestion. cheers, holger From lac at strakt.com Fri Oct 15 06:11:41 2004 From: lac at strakt.com (Laura Creighton) Date: Fri, 15 Oct 2004 06:11:41 +0200 Subject: [py-dev] Re: [py-svn] r6948 - in py/dist/py: . path/fspy path/pypath path/test test test/report test/report/text In-Reply-To: Message from hpk@codespeak.net of "Fri, 15 Oct 2004 05:57:10 +0200." <20041015035710.956575A0E7@thoth.codespeak.net> References: <20041015035710.956575A0E7@thoth.codespeak.net> Message-ID: <200410150411.i9F4BfAj012267@ratthing-b246.strakt.com> fspy looks like filesystem py to me. I liked the _name_ pypath better. fspython just looks like more typing to me. I think that this module ought to have widespread use indepent of the pypy project, so we should not saddle it with a second-best name because of possible confusion with better named (if more poorly implemented) thing. But I suspect we can brainstorm something better than pypath as well. I think the word 'root' belongs in there someplace? Other opinions? Laura > >I guess that naming it 'py.path.fspython' instead >of the short ..fspy may look nicer ?! > From hpk at trillke.net Fri Oct 15 11:54:56 2004 From: hpk at trillke.net (holger krekel) Date: Fri, 15 Oct 2004 11:54:56 +0200 Subject: [py-dev] Re: [py-svn] r6948 - in py/dist/py: . path/fspy path/pypath path/test test test/report test/report/text In-Reply-To: <200410150411.i9F4BfAj012267@ratthing-b246.strakt.com> References: <20041015035710.956575A0E7@thoth.codespeak.net> <200410150411.i9F4BfAj012267@ratthing-b246.strakt.com> Message-ID: <20041015095456.GD15456@solar.trillke.net> Hi Laura, [Laura Creighton Fri, Oct 15, 2004 at 06:11:41AM +0200] > fspy looks like filesystem py to me. That's actually what it is, a filesystem+python path to allow "addresses" of Python objects within a filesystem The name 'pypath' doesn't really hint at the filesystem-ness of this path implementations. anyway, at the moment i am just happy i changed the implementation and semantics of this little thingie because it will play a critical role for further py.test development because it allows to simply address python objects across process barriers. For running tests on multiple platforms and with multiple python versions having a way of unambigous talking about test methods is very important. Note though, that for most purposes you don't need to know about the existence of py.path.fspy() although i do want to export the object for the first major release's API. cheers, holger From lac at strakt.com Fri Oct 15 14:24:44 2004 From: lac at strakt.com (Laura Creighton) Date: Fri, 15 Oct 2004 14:24:44 +0200 Subject: [py-dev] Re: [py-svn] r6948 - in py/dist/py: . path/fspy path/pypath path/test test test/report test/report/text In-Reply-To: Message from hpk@trillke.net (holger krekel) of "Fri, 15 Oct 2004 11:54:56 +0200." <20041015095456.GD15456@solar.trillke.net> References: <20041015035710.956575A0E7@thoth.codespeak.net> <200410150411.i9F4BfAj012267@ratthing-b246.strakt.com> <20041015095456.GD15456@solar.trillke.net> Message-ID: <200410151224.i9FCOiGt013404@ratthing-b246.strakt.com> In a message of Fri, 15 Oct 2004 11:54:56 +0200, holger krekel writes: I appreciate that it is a path on a filesystem, but then I think that all paths are on filesystems. So that it is on a filesystem is not, to my mind, the most interesting thing about this, ass opposed to fsck(1) for instance, which really does want to operate on filesystems qua filesystems. Perhaps I am not making sense to you, or perhaps I am not understanding something about pypath -- or perhaps both, of course. Why is the filesystemness important enough to be worth mentioning? Laura >Hi Laura, > >[Laura Creighton Fri, Oct 15, 2004 at 06:11:41AM +0200] >> fspy looks like filesystem py to me. > >That's actually what it is, a filesystem+python path to >allow "addresses" of Python objects within a filesystem > >The name 'pypath' doesn't really hint at the filesystem-ness >of this path implementations. > >anyway, at the moment i am just happy i changed the >implementation and semantics of this little thingie because it >will play a critical role for further py.test development >because it allows to simply address python objects across >process barriers. For running tests on multiple platforms and >with multiple python versions having a way of unambigous >talking about test methods is very important. > >Note though, that for most purposes you don't need to know >about the existence of py.path.fspy() although i do want to >export the object for the first major release's API. > >cheers, > > holger From hpk at trillke.net Fri Oct 15 16:02:16 2004 From: hpk at trillke.net (holger krekel) Date: Fri, 15 Oct 2004 16:02:16 +0200 Subject: [py-dev] py.path.fspy / naming In-Reply-To: <200410151224.i9FCOiGt013404@ratthing-b246.strakt.com> References: <20041015035710.956575A0E7@thoth.codespeak.net> <200410150411.i9F4BfAj012267@ratthing-b246.strakt.com> <20041015095456.GD15456@solar.trillke.net> <200410151224.i9FCOiGt013404@ratthing-b246.strakt.com> Message-ID: <20041015140216.GJ15456@solar.trillke.net> Hi Laura, [Laura Creighton Fri, Oct 15, 2004 at 02:24:44PM +0200] > In a message of Fri, 15 Oct 2004 11:54:56 +0200, holger krekel writes: > > I appreciate that it is a path on a filesystem, but then I think that > all paths are on filesystems. So that it is on a filesystem is not, > to my mind, ... Good point to consider. The current py.path.* are indeed all filesystem based paths. But 'fspy' of all paths is actually more _and_ less than a filesystem path. On the one hand it is a "python module path" like e.g. 'os.path.abspath' and operations like dirpath() will yield something like 'os.path'. On the other hand it always starts from an explicit filesystem location unlike the python import system which starts somewhere hard-to-tell. Consequentely, you inintialize it with both a filesystemish path and a "python" path, therefore the name 'fspy' to make that clear. But you probably are right, that its filesystem-ness isn't the actual distinguishing feature. Hum. I guess i need to think some more about this ... a bientot (i also need to learn some french :-), holger From hpk at trillke.net Sun Oct 17 05:43:35 2004 From: hpk at trillke.net (holger krekel) Date: Sun, 17 Oct 2004 05:43:35 +0200 Subject: [py-dev] automatic documentation online Message-ID: <20041017034335.GD15456@solar.trillke.net> Hi, at http://codespeak.net/py/current/doc/ you'll find generated html files from the documentation text files at svn/py/dist/doc. This is automatic so whenever a commit happens it takes a few seconds on codespeak to show the latest. Note also, that if you already put something like eval `python ~/projects/py/py/env.py` into your shell boot script then you now have a little command line utility called 'rest.py' which is able to generate html files from ReST files which you specify on the command line. Without any arguments it does it in the current working dir. It will also link the html-docs with the 'style.css' file in the doc directory so that you can see the full documentation on your local side by pointing your browser to the doc-directory. and feel free to fix typos or add/correct stuff but please check with 'rest.py' in your doc directory that there are no ReST-errors. have fun, holger From david at sundayta.com Mon Oct 18 03:06:34 2004 From: david at sundayta.com (David Warnock) Date: Mon, 18 Oct 2004 02:06:34 +0100 Subject: [py-dev] Checkout problem Message-ID: <4173171A.7030901@sundayta.com> Hi, I like the progress and the new docs. So I tried to checkout to get started. [david at amos ~/dev/python]$ svn co http://codespeak.net/svn/py/dist dist-py svn: REPORT request failed on '/svn/!svn/vcc/default' svn: REPORT of '/svn/!svn/vcc/default': 400 Bad Request (http://codespeak.net) [david at amos ~/dev/python]$ I have not used subversion before and anyway should already be in bed so have given up for tonight. Any hints? Thanks Dave -- David Warnock: Sundayta Ltd. http://www.sundayta.com From hpk at trillke.net Mon Oct 18 03:14:28 2004 From: hpk at trillke.net (holger krekel) Date: Mon, 18 Oct 2004 03:14:28 +0200 Subject: [py-dev] Checkout problem In-Reply-To: <4173171A.7030901@sundayta.com> References: <4173171A.7030901@sundayta.com> Message-ID: <20041018011428.GK15456@solar.trillke.net> Hi David, [David Warnock Mon, Oct 18, 2004 at 02:06:34AM +0100] > Hi, > > I like the progress and the new docs. So I tried to checkout to get started. > > [david at amos ~/dev/python]$ svn co http://codespeak.net/svn/py/dist dist-py > svn: REPORT request failed on '/svn/!svn/vcc/default' > svn: REPORT of '/svn/!svn/vcc/default': 400 Bad Request > (http://codespeak.net) > [david at amos ~/dev/python]$ what does 'svn --version' tell? You should have an svn version of 1.0.x (1.0.5 or higher is best). cheers, holger From david at sundayta.com Mon Oct 18 10:32:55 2004 From: david at sundayta.com (David Warnock) Date: Mon, 18 Oct 2004 09:32:55 +0100 Subject: [py-dev] Checkout problem In-Reply-To: <20041018011428.GK15456@solar.trillke.net> References: <4173171A.7030901@sundayta.com> <20041018011428.GK15456@solar.trillke.net> Message-ID: <41737FB7.6020602@sundayta.com> holger, > what does 'svn --version' tell? > > You should have an svn version of 1.0.x (1.0.5 or higher is best). Looks like that is ok, see below. Thanks Dave [david at amos ~/dev/python]$ svn --version svn, version 1.0.5 (r9954) compiled Jun 10 2004, 16:45:17 Copyright (C) 2000-2004 CollabNet. Subversion is open source software, see http://subversion.tigris.org/ This product includes software developed by CollabNet (http://www.Collab.Net/). The following repository access (RA) modules are available: * ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol. - handles 'http' schema - handles 'https' schema * ra_svn : Module for accessing a repository using the svn network protocol. - handles 'svn' schema -- David Warnock: Sundayta Ltd. http://www.sundayta.com From hpk at trillke.net Mon Oct 18 11:06:23 2004 From: hpk at trillke.net (holger krekel) Date: Mon, 18 Oct 2004 11:06:23 +0200 Subject: [py-dev] Checkout problem In-Reply-To: <41737FB7.6020602@sundayta.com> References: <4173171A.7030901@sundayta.com> <20041018011428.GK15456@solar.trillke.net> <41737FB7.6020602@sundayta.com> Message-ID: <20041018090623.GM15456@solar.trillke.net> [David Warnock Mon, Oct 18, 2004 at 09:32:55AM +0100] > holger, > > >what does 'svn --version' tell? > > > >You should have an svn version of 1.0.x (1.0.5 or higher is best). > > Looks like that is ok, see below. hum, can you retry with svn co http://codespeak.net:8080/svn/py/dist dist-py to see if some http-proxy is getting in your way? holger From david at sundayta.com Mon Oct 18 11:46:04 2004 From: david at sundayta.com (David Warnock) Date: Mon, 18 Oct 2004 10:46:04 +0100 Subject: [py-dev] Checkout problem In-Reply-To: <20041018090623.GM15456@solar.trillke.net> References: <4173171A.7030901@sundayta.com> <20041018011428.GK15456@solar.trillke.net> <41737FB7.6020602@sundayta.com> <20041018090623.GM15456@solar.trillke.net> Message-ID: <417390DC.3080508@sundayta.com> holger, > hum, can you retry with > > svn co http://codespeak.net:8080/svn/py/dist dist-py > > to see if some http-proxy is getting in your way? Perfect, sorry I should have thought of that myself. I have an IPCOP firewall with Squid running as a transparent proxy. Many thanks. I have a proper copy now. Dave -- David Warnock: Sundayta Ltd. http://www.sundayta.com From hpk at trillke.net Mon Oct 18 12:15:36 2004 From: hpk at trillke.net (holger krekel) Date: Mon, 18 Oct 2004 12:15:36 +0200 Subject: [py-dev] Checkout problem In-Reply-To: <417390DC.3080508@sundayta.com> References: <4173171A.7030901@sundayta.com> <20041018011428.GK15456@solar.trillke.net> <41737FB7.6020602@sundayta.com> <20041018090623.GM15456@solar.trillke.net> <417390DC.3080508@sundayta.com> Message-ID: <20041018101536.GR15456@solar.trillke.net> Hi David, [David Warnock Mon, Oct 18, 2004 at 10:46:04AM +0100] > holger, > > >hum, can you retry with > > > > svn co http://codespeak.net:8080/svn/py/dist dist-py > > > >to see if some http-proxy is getting in your way? > > Perfect, sorry I should have thought of that myself. I have an IPCOP > firewall with Squid running as a transparent proxy. > > Many thanks. I have a proper copy now. great, have fun. If you have any problems, don't hesitate to ask. You can easily get an account on codespeak if you want to commit fixes ... and please note that the documentation is still somewhat ahead of the code in some respects ... but i am catching up and especially need to press Armin on considering the new import/export scheme some more, see the future document: http://codespeak.net/py/current/doc/future.html That's the next big thing because then it would be possible to write a nice documentation tool that will generate documentation for the complete py-API, cross-referenced from the ReST-documentation ... another chapter in the future book :-) cheers, holger From hpk at trillke.net Tue Oct 19 17:43:30 2004 From: hpk at trillke.net (holger krekel) Date: Tue, 19 Oct 2004 17:43:30 +0200 Subject: [py-dev] Re: [py-svn] r7025 - in py/dist/py: path path/extpy path/local path/svn test In-Reply-To: <20041019134021.3CA045A15E@thoth.codespeak.net> References: <20041019134021.3CA045A15E@thoth.codespeak.net> Message-ID: <20041019154330.GR15456@solar.trillke.net> Hi Armin, [arigo at codespeak.net Tue, Oct 19, 2004 at 03:40:21PM +0200] > New Revision: 7025 > ... > Log: > Get rid of the imp module to import local modules. Instead: > > FSPathBase.getmodule(): uses a new internal method 'self.getcodeobj()' to get > the module's top-level code object, and then (as before) build a module > object, put it in sys.modules, and run the code in the module. > > FSPathBase.getcodeobj(): uses self.read() and dyncode as before. > > LocalPath.getcodeobj(): uses the regular compile() to make the code object, so > that a local module shows up normally in tracebacks and in the co_filename > attributes. This is also where we look for and create .pyc files. In this > scheme, .pyc files are completely ignored for non-local paths. This is > probably not too bad: with svn paths, say, creating the .pyc files would add > them to the repository. > > Plus a few details, including setting up the new module's __path__ so that > relative imports work (at least on local paths). all nice! But i think we should signal more clearly that these methods refer to *python* files. Or do it slightly differently by, for example, providing a py.magic.View and defining the "getpymodule()" and "getpycode()" on a separated out layer to reduce namespace complexity. I am not completly sure though that this is feasible. At least we should rename the methods to getpycode/getpymodule. cheers, holger From hpk at trillke.net Wed Oct 20 03:53:14 2004 From: hpk at trillke.net (holger krekel) Date: Wed, 20 Oct 2004 03:53:14 +0200 Subject: [py-dev] transparent compilation / greenlets? Message-ID: <20041020015314.GX15456@solar.trillke.net> hi, i just added a "transparent compilation" section (aka "crossing the barriers of compilation") to the future chapter of the current documentation: http://codespeak.net/py/current/doc/future.html and i wonder, does anyone here have an interesting C-module to offer which would be nice to integrate into the py lib? And what about greenlets anyway? :-) I'd like to experiment a bit with them and we could mark them as experimental if we are not sure we want to continue to have them. Hum, i guess it's pretty clear where they would go: py.magic.greenlet :-) This would be a worthwhile test case for getting nice "transparent compilation". cheers, holger From hpk at trillke.net Fri Oct 29 15:35:01 2004 From: hpk at trillke.net (holger krekel) Date: Fri, 29 Oct 2004 15:35:01 +0200 Subject: [py-dev] utest thoughts In-Reply-To: <41604DE5.8080703@colorstudy.com> References: <4158A934.4040800@colorstudy.com> <20041001164412.GN7311@solar.trillke.net> <415D9C6C.5000702@colorstudy.com> <20041003114016.GA7311@solar.trillke.net> <41604DE5.8080703@colorstudy.com> Message-ID: <20041029133501.GP15456@solar.trillke.net> Hi Ian, i have a second thought regarding one of your suggestions/comment regarding custom test collectors ... [Ian] > [me] > >b) collectors always yield collectors or Units, not plain functions > > so you would actually do > > > > yield py.test.Unit(my_custom_test_function, *args) > > > >In the current py code (see my other postings) you'll find > >in py/test/test/data/Collector.py the current way to do custom > >collectors. This Collector class is instantiated with > >the module's location and takes over the collecting > >process for the module. No further attempt is made > >to collect anything from a module which contains a > >custom "Collector" object. > > I have a problem with APIs that jump in complexity. In this case you > can forget about Unit until you start needing to generate multiple tests > programmatically -- your functions *were* your units, until that point. > If functions are units in one place, they should be units everyplace. I agree that a jump in complexity needs to be justified and for the discussed case it's arguable: Custom collectors or generating custom tests should be easy and if possible a "no-API" thing. Today i thought that maybe it isn't such a bad idea to allow generative tests which would come close to what collectors provide. def test_generating(self): for x in someiter: yield x would yield new test methods/objects which would be wrapped in an appropriate Item-class in order to allow the reporter to uniformly access information about test items. The py.test.Driver could detect "generating tests" by checking if ``test_obj()`` returns a generator or if ``iter(test_obj)`` succeeds. This, however, would defer the decision if something is a collector aka a "generative test" or a plain test item to a very late point which leads to the the question if the distinction between collectors and test items can't be removed altogether. I think i am going to give this a try soon. Comments welcome. cheers, holger