[pytest-dev] [pytest-rerunfailures] pytest-rerunfailures not using fixtures on reruns (#10)

[holger krekel]

Hey Leah,

On Mon, Apr 22, 2013 at 15:08 -0700, Leah Klearman wrote:
> Hey Holger,
> 
> Thank you for the timely patch. I haven't seen Bob online today, but
> hopefully he'll be able to test it very soon.
> 
> A word of warning: your calling of runtestprotocol() is not quite right
> > and might lead to problems.  "nextitem" should really be the item that
> > is going to be run next.  So if you re-run three times the first two
> > invocations should have nextitem=item.
> 
> 
> I agree that that would be ideal, but it only re-runs the test if it fails,
> so I
> would have to know the future in order to know what value to send nextitem=.
> Since hopefully most tests pass the first try, sending nextitem instead of
> item seems to be the better answer.

FYI pytest perform teardown/finalization according to nextitem:  if your
next item uses different fixtures, the "item" ones may be torn down
(it also depends on the caching scope of the fixtures).

Do you generally want to re-run the setup or could you also settle 
on just re-running the "call" phase of a test?  The latter one would
be easier and probably not disrupt fixture mechanics. 

best,
holger

> Thanks!
> -Leah
> 
> 
> On Mon, Apr 22, 2013 at 1:27 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hi Leah, Bob,
> >
> > On Fri, Apr 19, 2013 at 20:55 -0700, Leah Klearman wrote:
> > > Hi Holger and other py.test mavens,
> > >
> > > Bob has reported a problem with my py.test plugin pytest-rerunfailures
> > [1]
> > > not re-running the setup before rerunning the test.
> > >
> > > Looking at my code, it has pytest_runtest_protocol() [2] looping
> > > on _pytest.runner.runtestprotocol() [3], which in turn runs the setup,
> > the
> > > test, and the teardown.
> > >
> > > [1] https://github.com/klrmn/pytest-rerunfailures
> > > [2]
> > >
> > https://github.com/klrmn/pytest-rerunfailures/blob/master/rerunfailures/plugin.py#L46
> > > [3]
> > >
> > https://bitbucket.org/hpk42/pytest/src/fdc28ac2029f1c0be1dac4991c2f1b014c39a03f/_pytest/runner.py?at=default#cl-65
> > >
> > > I haven't taken the hours needed to get my head fully into py.test plugin
> > > development mode, but I'm not sure I can implement a fix at my layer.
> > >
> > >
> > > I'm hoping someone here will have some insight.
> >
> > It's a bit intricate.  A function item keeps around some fixture state
> > and was so far not intended to be run multiple times.  I went ahead and
> > tried to improve the behaviour to better allow re-running.   Please try
> > with
> >
> >     pip install -i http://pypi.testrun.org -U pytest
> >
> > which should give you pytest-2.3.5.dev16 at least.  This is bound to be
> > released soon so quick feedback is welcome.  If you still have problems
> > please try to send a minimal test file which shows undesired behaviour.
> >
> > A word of warning: your calling of runtestprotocol() is not quite right
> > and might lead to problems.  "nextitem" should really be the item that
> > is going to be run next.  So if you re-run three times the first two
> > invocations should have nextitem=item.
> >
> > best,
> > holger
> >
> >
> > > Thanks,
> > > -Leah
> > >
> > >
> > > On Sun, Apr 14, 2013 at 6:29 PM, Bob Silverberg <
> > notifications at github.com>wrote:
> > >
> > > > I just verified this behaviour myself with a simple test [1]. I see it
> > > > with both funcargs and fixtures, but I'm not sure if it has to do with
> > the
> > > > plugin, or the way py.test works. It does inject the value into the
> > test
> > > > method, but it doesn't rerun the fixture, so it seems like it is
> > caching
> > > > the first run of the fixture and using that on subsequent runs.
> > > >
> > > > I'm not sure if this is something that the plugin can have any effect
> > on,
> > > > or if it's just the way fixtures work. It is specified for this fixture
> > > > that it is scope='function', and perhaps py.test makes that happen by
> > > > checking the function name, which is, of course, the same for each
> > run. I
> > > > did try removing the scope argument from the fixture but that had no
> > > > effect.
> > > >
> > > > Do you have any thoughts about this, @klrmn <https://github.com/klrmn
> > >?
> > > >
> > > > [1] https://gist.github.com/bobsilverberg/5385035
> > > >
> > > > —
> > > > Reply to this email directly or view it on GitHub<
> > https://github.com/klrmn/pytest-rerunfailures/issues/10#issuecomment-16363644
> > >
> > > > .
> > > >
> >
> > > _______________________________________________
> > > Pytest-dev mailing list
> > > Pytest-dev at python.org
> > > http://mail.python.org/mailman/listinfo/pytest-dev
> >
> >