[Python-Dev] please help triage VSTS failures

[Gregory P. Smith]

Hah, yes, that was un-intuitive for me.  I was looking for something
labelled "logs" to click on.  thanks.  (I hope MS is taking notes on UX
issues here)

So these failures were in fact the known flakes and not infrastructure.
good!

On the high level view of VSTS output on a failure:

"Issues:
phase1
Cmd.exe exited with code '2'."

is not useful.  Can we make that display the tail of the failing phase's
log?  that'd avoid needing to find the inner most red X and click on it
which is what I'll always need to do otherwise as there doesn't appear to
be a direct hyperlink to that.

It looks like VSTS also had an API for surfacing individual test results
into their Tests / Test Results summary pane?  Doing something to integrate
with that would likely be a nicer UI.  We have something similar at work,
here's how we make unittest.TestCase emit the JUnit XML files of test
results for CI systems to display:

https://github.com/abseil/abseil-py/blob/master/absl/testing/xml_reporter.py


If there are industry standard format(s) that make Python test suites play
nicer for reporting on CI systems such as that, they'd be worthy of stdlib
inclusion.

-gps


On Fri, May 18, 2018 at 5:35 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, May 18, 2018 at 1:13 PM, Steve Dower <steve.dower at python.org>
> wrote:
> > According to the VSTS dev team, an easy “rerun this build” button and
> > filtering by changed paths are coming soon, which should clean things up.
>
> If you're talking to them, please ask them to make sure that the
> "rerun this build" button doesn't erase the old log. (That's what it
> does on Travis. Appveyor is better.) The problem is that when you have
> a flaky/intermittent failure, your todo list is always (a) rerun the
> build so at least it's not blocking whatever this unrelated change is,
> (b) file some sort of bug, or comment on some existing bug, and link
> to the log to help track down the intermittent failure. If you click
> the "rebuild" button on Travis, then it solves (a), while deleting the
> information you need for (b) – and for rare intermittent bugs you
> might not have much information to go on besides that build log.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180518/9ff7419b/attachment.html>