[pytest-dev] [pytest-rerunfailures] pytest-rerunfailures not using fixtures on reruns (#10)

[Leah Klearman]

Holger,

The only reason I can think of why py.test would perform teardown /
finalization differently depending on the nextitem is if class, module, and
package level fixtures are being used. If only method level fixtures are
used, then they should always get created and always get torn down for each
test case. Is that not how it works?

We can't re-run just the 'call' phase because the failure of the previous
'call' phase may have left behind a partial situation (see automated
testing before we had setup and teardown and fixtures).

-Leah


On Tue, Apr 23, 2013 at 12:16 AM, holger krekel <holger at merlinux.eu> wrote:

> Hey Leah,
>
> On Mon, Apr 22, 2013 at 15:08 -0700, Leah Klearman wrote:
> > Hey Holger,
> >
> > Thank you for the timely patch. I haven't seen Bob online today, but
> > hopefully he'll be able to test it very soon.
> >
> > A word of warning: your calling of runtestprotocol() is not quite right
> > > and might lead to problems.  "nextitem" should really be the item that
> > > is going to be run next.  So if you re-run three times the first two
> > > invocations should have nextitem=item.
> >
> >
> > I agree that that would be ideal, but it only re-runs the test if it
> fails,
> > so I
> > would have to know the future in order to know what value to send
> nextitem=.
> > Since hopefully most tests pass the first try, sending nextitem instead
> of
> > item seems to be the better answer.
>
> FYI pytest perform teardown/finalization according to nextitem:  if your
> next item uses different fixtures, the "item" ones may be torn down
> (it also depends on the caching scope of the fixtures).
>
> Do you generally want to re-run the setup or could you also settle
> on just re-running the "call" phase of a test?  The latter one would
> be easier and probably not disrupt fixture mechanics.
>
> best,
> holger
>
> > Thanks!
> > -Leah
> >
> >
> > On Mon, Apr 22, 2013 at 1:27 AM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > > Hi Leah, Bob,
> > >
> > > On Fri, Apr 19, 2013 at 20:55 -0700, Leah Klearman wrote:
> > > > Hi Holger and other py.test mavens,
> > > >
> > > > Bob has reported a problem with my py.test plugin
> pytest-rerunfailures
> > > [1]
> > > > not re-running the setup before rerunning the test.
> > > >
> > > > Looking at my code, it has pytest_runtest_protocol() [2] looping
> > > > on _pytest.runner.runtestprotocol() [3], which in turn runs the
> setup,
> > > the
> > > > test, and the teardown.
> > > >
> > > > [1] https://github.com/klrmn/pytest-rerunfailures
> > > > [2]
> > > >
> > >
> https://github.com/klrmn/pytest-rerunfailures/blob/master/rerunfailures/plugin.py#L46
> > > > [3]
> > > >
> > >
> https://bitbucket.org/hpk42/pytest/src/fdc28ac2029f1c0be1dac4991c2f1b014c39a03f/_pytest/runner.py?at=default#cl-65
> > > >
> > > > I haven't taken the hours needed to get my head fully into py.test
> plugin
> > > > development mode, but I'm not sure I can implement a fix at my layer.
> > > >
> > > >
> > > > I'm hoping someone here will have some insight.
> > >
> > > It's a bit intricate.  A function item keeps around some fixture state
> > > and was so far not intended to be run multiple times.  I went ahead and
> > > tried to improve the behaviour to better allow re-running.   Please try
> > > with
> > >
> > >     pip install -i http://pypi.testrun.org -U pytest
> > >
> > > which should give you pytest-2.3.5.dev16 at least.  This is bound to be
> > > released soon so quick feedback is welcome.  If you still have problems
> > > please try to send a minimal test file which shows undesired behaviour.
> > >
> > > A word of warning: your calling of runtestprotocol() is not quite right
> > > and might lead to problems.  "nextitem" should really be the item that
> > > is going to be run next.  So if you re-run three times the first two
> > > invocations should have nextitem=item.
> > >
> > > best,
> > > holger
> > >
> > >
> > > > Thanks,
> > > > -Leah
> > > >
> > > >
> > > > On Sun, Apr 14, 2013 at 6:29 PM, Bob Silverberg <
> > > notifications at github.com>wrote:
> > > >
> > > > > I just verified this behaviour myself with a simple test [1]. I
> see it
> > > > > with both funcargs and fixtures, but I'm not sure if it has to do
> with
> > > the
> > > > > plugin, or the way py.test works. It does inject the value into the
> > > test
> > > > > method, but it doesn't rerun the fixture, so it seems like it is
> > > caching
> > > > > the first run of the fixture and using that on subsequent runs.
> > > > >
> > > > > I'm not sure if this is something that the plugin can have any
> effect
> > > on,
> > > > > or if it's just the way fixtures work. It is specified for this
> fixture
> > > > > that it is scope='function', and perhaps py.test makes that happen
> by
> > > > > checking the function name, which is, of course, the same for each
> > > run. I
> > > > > did try removing the scope argument from the fixture but that had
> no
> > > > > effect.
> > > > >
> > > > > Do you have any thoughts about this, @klrmn <
> https://github.com/klrmn
> > > >?
> > > > >
> > > > > [1] https://gist.github.com/bobsilverberg/5385035
> > > > >
> > > > > —
> > > > > Reply to this email directly or view it on GitHub<
> > >
> https://github.com/klrmn/pytest-rerunfailures/issues/10#issuecomment-16363644
> > > >
> > > > > .
> > > > >
> > >
> > > > _______________________________________________
> > > > Pytest-dev mailing list
> > > > Pytest-dev at python.org
> > > > http://mail.python.org/mailman/listinfo/pytest-dev
> > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20130423/a24c2d61/attachment.html>