[py-dev] Issue with py.test and stderr

[Grig Gheorghiu]

Holger,

Thanks for the detailed response. I'm still very much green at py.test,
but I'm working on it :-)

pyUnitPerf doesn't actually do profiling, it does load/time performance
testing. You can start by profiling your code for bottlenecks via the
profile module for example, and then wrap those bottleneck
functions/methods in unitperf tests. You can set a desired user load
and an expected time to completion, and have the tests fail if the time
is exceeded. There's a good article by Mike Clark at
http://javapronews.com/javapronews-47-20030721ContinuousPerformanceTestingwithJUnitPerf.html

I think the notion of "continuous performance testing" is worth
investigating, so that's what I was referring to when I said I'll give
some thought to a potential py.test.perf module. 

Grig

--- holger krekel <hpk at trillke.net> wrote:

> Hi Grig!
> 
> On Fri, Jan 14, 2005 at 11:56 -0800, Grig Gheorghiu wrote:
> > I started to play with py.test after seeing it mentioned on a lot
> of
> > Python-related blogs. I find it really easy to use and powerful at
> the
> > same time, so congrats for a fine piece of work.
> 
> great to hear! 
> 
> > I did run into a slight issue though. At some point I was
> instantiating
> > a class which was redirecting stderr to stdout under some
> > circumstances. The __del__ method of my class was resetting
> sys.stderr
> > to sys.__stderr__. I think this caused a problem with py.test, in
> that
> > it interfered with py.test's own manipulation of sys.stderr. 
> 
> Yes, certainly.  The stdout/stderr capturing is a bit naive and not
> careful enough yet.  Note that if you suspect that py.test messes up 
> stdout/stderr somehow you can pass 
> 
>     py.test -S 
> 
> to inhibit the redirecting logic. 
> 
> > The end result was that I was getting a rather cryptic error
> message
> > from py.test:
> 
> > ... traceback too ugly to cite here ... 
> 
> uh indeed :-) 
> 
> > The only way I was able to get around this issue without modifying
> my
> > class was to edit py/test/tool/outerrcapture.py and replace:
> > 
> > err = e.getvalue()
> > 
> > with 
> > 
> > try:
> >     err = e.getvalue()
> > except:
> >     err = None
> >
> > Then things seemed to work fine. 
> 
> "Do the simplest fix that possibly works" .. .
> aah these testing guys :-) 
> 
> Well, in this case we should probably do better and detect
> that a test did something to stdout/stderr.  The test reporter
> could display something like 
> 
>     [test changed stdout/err -> partial output only] 
> 
> and still display what we captured.  It may generally 
> be a good idea to always reset stdout/stderr after a 
> test ran. 
> 
> > P.S. I'm really interested in both Python and testing, so if you
> guys
> > need some help with py.test in the future, I'd be glad to pitch in.
> I
> > have a blog with some of the stuff I'm doing at
> > http://agiletesting.blogspot.com . 
> 
> Interesting.  
> 
> > I've been using unittest quite heavily, but I'm also trying to look
> > into alternatives, mainly py.test and doctest.
> 
> doctests are to be integrated into py.test real soon now 
> (this is a programmer saying this!). 
> 
> Actually the Schevo guys did a hack in their tree 
> and i have doing some tries with Armin.  I think the
> main issues are how to collect and report doctests. 
> As you noticed, py.test is so far collecting test 
> automatically. For doctests this may not be completly 
> feasible. 
> 
> > I released a JUnitPerf
> > port to Python which I called pyUnitPerf, so now I'm thinking about
> > doing something similar and more "pythonic" with py.test. Maybe
> > py.test.perf?!! 
> 
> Profiling certainly makes sense in the testing context.  It's not 
> too high on my list, though.  But if you want to give it a try, 
> that would be great. I could certainly help with integrational 
> issues. 
> 
> cheers, 
> 
>     holger
>