[py-dev] using funcargs for setup/teardown

[Philippe Fremy]

Hi Holger,

Same mail again, this time in plain good old text. Sorry for that, now
that I'm using only html for professional email, I tend to forget my
good manners for open source projects.


holger krekel wrote:
> Hi Philippe,
> 
> On Wed, Oct 28, 2009 at 08:09 +0100, Philippe Fremy wrote:
> 
>> holger krekel wrote:
>> 
>>> Hi Frederik!
>>> 
>>> 
>>>> However, I've finally figured out why I'm having such a hard 
>>>> time warming up to funcargs: They go way beyond the minimalist
>>>>  simplicity that made me switch to py.test in the first place.
>>>>  (I realize there's some Magic to py.test's internals, but that
>>>>  doesn't surface in the API.)
>>>> 
>>>> Funcargs seem like a departure from this principle. I 
>>>> understand that some situations demand such complexity, so I'm
>>>>  not arguing against funcargs in general - but for my part,
>>>> I've managed to keep it simple so far.
>>>> 
>>>> 
>> I second Frederic here. I chose py.test because of the overall 
>> simplicity of the framework when you use it. The 
>> setup_module/setup_class/setup_method without any special magic 
>> other than using function, class or method names starting with test
>>  is really really nice. I am kind of worried because I could no 
>> longer find them in the documentation. I eventually found them in 
>> the xUnit documentation but that's not where I expected them. I 
>> vote for a return of this documentation to the main py.test 
>> documentation, and moving funcarg related documentation to an 
>> "advanced test setup" section.
>> 
> 
> I am worried that newcomers will be confused then.  But i see the 
> point of "unhiding" the xUnit documentation.
> 

You could present the two approaches in the documentation and explaining
what are the differences.

The simplicity of the setup/teardown model is really a strong asset for
testing. It makes the test code architecture understandable by people
not familiar with py.test . The same can not be said for the funcarg
mechanism...

>> funcarg seems to be a powerful tool, but really cumbersome to grok.
>>  I find the magic trick on the naming with __myargument a bit 
>> cumbersome. I would feel more comfortable with a syntax that mimic
>>  the setup/teardown used for module, class and methods.
>> 
> 
> It would be easy to also allow explicit registration for factories 
> ala:
> 
> def pytest_configure(config): config.add_funcarg_factory(name, 
> factoryfunc)
> 
> but at the time i choose "convention over configuration" and the "__"
>  because Django also uses a similar way to encode values into a 
> function name.
> 

I can understand that, but it does not "speak" to people that are not
famliar with this approach. By using this convention, my impression is
that you are leaving behind the people who are just used to functions,
classes and methods.


>>> This implies having to call magic methods for setting up objects
>>>  in global namespaces or 'self' attributes - which not only makes
>>>  the test harder to understand and refactor IMO.
>>> 
>> I do not agree here. If I have one class with 30 unit test methods,
>>  it's easier to setup/teardown the test parameters in two methods 
>> for the whole class than modifying 30 test methods to add funcargs
>>  arguments.
>> 
> 
> agreed, i am considering introducing a new hook:
> 
> def pytest_pyfunc_setup(request): request.cached_setup(setupfunc, 
> teardownfunc, scope="directory")
> 
> and this hook would be called for each python test function.  Here we
>  call the cached_setup helper to help us manage setup/teardown 
> scopes. (see http://tinyurl.com/yfw82l5 for request object attributes
>  which you could pass into your setup/teardown func)
> 
> It would also mean you could write down:
> 
> class TestGroup: parameter = 3 def test_method(self): assert 
> self.parametrized_obj  ...
> 
> and implement a pytest_pyfunc_setup() to set a "self.someobj" 
> according to the class-specified parameter.
> 
> Would this make sense to you?
> 

Honestly, I am not sure I get it all.  Can you sketch full examples ?

By the way, is it possible to dynamically update the py.test namespace ?
There is the initial hook py_test_namespace() but it's only called once.
A py.test.update_namespace() would be welcome in my case.

>> So, while I agree that funcargs certainly has potential, I think 
>> you should not force it onto the user and should really stress the
>>  two ways of setting up per class or per method parameters.
>> 
> 
> ok, I'll see to work on the docs a bit.
> 
> 
>> And it would be really nice to figure out a syntax for funcarg that
>>  is more in the setup/teardown fashion.
>> 
>> I don't see much gain of using :
>> 
>> def pytest_funcarg__mysetup(request): return MySetup(request)
>> 
>> class TestClass: def test_function(self, mysetup): conn = 
>> mysetup.getsshconnection() # work with conn
>> 
>> 
>> instead of :
>> 
>> class TestClass: def setup_class( c ): c.mysetup = MySetup()
>> 
>> def test_function(self): conn = c.mysetup.getsshconnection() # work
>>  with conn
>> 
>> 
>> The programming style is different, the second one is traditional 
>> OO and will be familiar to anybody coming from C++, Java or other 
>> OO world. The second one is playing more on the python 
>> capabilities. I don't find it more readable.
>> 
> 
> If you have setup_module + setup_class + setup_method (+ 
> subclassing!) layers of setup and teardown, things get a bit harder 
> to understand. In >3000 test projects i know off this led to 
> complexity and refactoring difficulties.
> 

I haven't reached for > 3000 test cases, so I can not comment on that.
But for small projects like the one I am working with (around a few
hundreds tests), the setup/teardown works very well.

>> One can argue about the advantage that each test function can take
>>  a different parameter. While true, in my testing experience, I 
>> haven't seen a pattern with many different tests taking many 
>> different parameters. I usually have groups of 5 to 10 tests taking
>>  one kind of parameter. If they need to take another kind of 
>> parameter, I will put them in a different test class.
>> 
> 
> These days i sometimes group tests by feature rather than by
> setup-parameters. E.g. i put two unit-tests and one functional test
> into a "TestFeature" class.
> 

Then you need a different setup depending on which test you are running
indeed. But you can already do that by discriminating on the method
argument of the setup_method(), no ?

>> So, in my opinion, using funcargs for regular cases is a matter of
>>  style.
>> 
>> I do see a big value in funcargs for the parameterized tests. 
>> Running the same test over and over with different parameter is a 
>> really nice feature. I think you should stress it more in the 
>> documentation.
>> 
> 
> makes sense. thanks a lot for your feedback!
> 
> best, holger
> 
>