[docs] unittest: test seem to be run out of order

[Dwyer, Chris (KBB)]

Hello,

I am new to unittest, so I may have missed something in the docs.  My understanding was that tests were run in the order they appear in the class.

I am writing a series of tests that are related and I named the tests in the following manor:
Testarea test case number test case letter (number major test area relation, letter minor test area relation)

I wanted something like:

California01a
California01b
California02a
California02b
California02c
California03a
California03b
California04a
California04b
California04c
California05a
California05b
California06a
California06b
California06c
California07a
California07b
California08a
California08b
California08c

However I had a typo and ended up with this:

California01a
California01b
California02a
California02b
California02c
California03a
California03b
California04a
California04b
California04c
California05a
California05b
California6a
California06b
California06c
California07a
California07b
California08a
California08b
California08c

The first run seemed to run the California6a last.

I corrected the typo and reran the test and everything ran in the expect order.

Here is the output I capture for the failed run:
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict02a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL
test_YMM_409_Conflict02b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... ok
test_YMM_409_Conflict02c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... FAIL
test_YMM_409_Conflict03a (__main__.Test_YMM_Core)
Verify error code - "409" ... ok
test_YMM_409_Conflict03b (__main__.Test_YMM_Core)
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict04a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL
test_YMM_409_Conflict04b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... ok
test_YMM_409_Conflict04c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... FAIL
test_YMM_409_Conflict05a (__main__.Test_YMM_Core)
Verify error code - "409" ... ok
test_YMM_409_Conflict05b (__main__.Test_YMM_Core)
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict06b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... FAIL
test_YMM_409_Conflict06c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... ERROR
test_YMM_409_Conflict07a (__main__.Test_YMM_Core)
Verify error code - "409" ... ok
test_YMM_409_Conflict07b (__main__.Test_YMM_Core)
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict08a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL
test_YMM_409_Conflict08b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... ok
test_YMM_409_Conflict08c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... FAIL
test_YMM_409_Conflict6a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL

======================================================================
ERROR: test_YMM_409_Conflict06c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests


Here is the output from the run that did not get an error:
test_YMM_409_Conflict01a (__main__.Test_YMM_Core)
Verify error code - "409" ... ok
test_YMM_409_Conflict01b (__main__.Test_YMM_Core)
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict02a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL
test_YMM_409_Conflict02b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... ok
test_YMM_409_Conflict02c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... FAIL
test_YMM_409_Conflict03a (__main__.Test_YMM_Core)
Verify error code - "409" ... ok
test_YMM_409_Conflict03b (__main__.Test_YMM_Core)
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict04a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL
test_YMM_409_Conflict04b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... ok
test_YMM_409_Conflict04c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... FAIL
test_YMM_409_Conflict05a (__main__.Test_YMM_Core)
Verify error code - "409" ... ok
test_YMM_409_Conflict05b (__main__.Test_YMM_Core)
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict06a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL
test_YMM_409_Conflict06b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... ok
test_YMM_409_Conflict06c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... FAIL
test_YMM_409_Conflict07a (__main__.Test_YMM_Core)
Verify error code - "409" ... ok
test_YMM_409_Conflict07b (__main__.Test_YMM_Core)
Verify error message - "Requested category requires inclusion of a valid identifier" ... ok
test_YMM_409_Conflict08a (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Failed_Insight_Requests ... FAIL
test_YMM_409_Conflict08b (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Request_Duration ... ok
test_YMM_409_Conflict08c (__main__.Test_YMM_Core)
Verify telemetry counter ...InsightTelemetryCounters.Total_Insight_Requests ... FAIL

Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/docs/attachments/20150828/16b65e17/attachment-0001.html>