[Python-Dev] Unittest/doctest formatting differences in 2.7a1?

[exarkun at twistedmatrix.com]

On 9 Dec, 06:09 pm, fuzzyman at voidspace.org.uk wrote:
>On 09/12/2009 18:02, exarkun at twistedmatrix.com wrote:
>>On 05:11 pm, lregebro at jarn.com wrote:
>>>On Wed, Dec 9, 2009 at 17:34, Michael Foord 
>>><fuzzyman at voidspace.org.uk> wrote:
>>>>Can you be more specific?
>>>
>>>Only with an insane amount of work. I'll hold that off for a while.
>>
>>I don't know if this is related at all (and I guess we won't until 
>>Lennart can be more specific :), but here are some Twisted unit test 
>>failures which are probably due to unittest changes in 2.7:
>>
>>===============================================================================
>>[FAIL]: twisted.trial.test.test_loader.LoaderTest.test_sortCases
>>
>>Traceback (most recent call last):
>>  File 
>>"/home/buildslave/pybot/trunk.glass-x86/build/Twisted/twisted/trial/test/test_loader.py", 
>>line 167, in test_sortCases
>>    [test._testMethodName for test in suite._tests])
>>twisted.trial.unittest.FailTest: not equal:
>>a = ['test_b', 'test_c', 'test_a']
>>b = ['test_c', 'test_b', 'test_a']
>
>Is the order significant? Can you just compare sorted versions of the 
>lists instead?
>
>If the order *is* significant I would be interested to know which 
>change caused this.

It looks like a change to shortDescription may be responsible for all 
the failures mentioned here.

The order *is* significant (it's a test for how tests are ordered...). 
The sorting code (which wasn't super awesome, I'll admit) it uses was 
broken by the change in the return value of shortDescription, something 
which is much more obvious in some of the other failures.
>[snip]
>>a = 'skip1 (twisted.trial.test.test_tests.SkippingTests)'
>>b = 'skip1'
>
>These two test failures are due to the change in repr of something I 
>guess? Is a or b the original output?

b is the original, a is the produced value.  Here's a simpler example:

Python 2.6.4 (r264:75706, Nov  2 2009, 14:38:03) [GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>from twisted.test.test_abstract import AddressTests
>>>AddressTests('test_decimalDotted').shortDescription()
'test_decimalDotted'
>>>

Python 2.7a0 (trunk:76325M, Nov 16 2009, 09:50:40) [GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>from twisted.test.test_abstract import AddressTests
>>>AddressTests('test_decimalDotted').shortDescription()
'test_decimalDotted (twisted.test.test_abstract.AddressTests)'
>>>

Aside from compatibility issues, this seems like a not-too-ideal change 
in behavior for a method named "shortDescription", at least to me. 
Going out on a limb, I'll guess that it was made in order to provide a 
different user-facing experience in the stdlib's test runner.  If that's 
right, then I think it would make sense to revert shortDescription back 
to the previous behavior and either introduce a new method which returns 
this string or just put the logic all into the display code instead.
>>
>>I'm not opposed to the improvement of unittest (or any part of 
>>Python). Perhaps more of the improvements can be provided in new APIs 
>>rather than by changing the behavior of existing APIs, though.
>
>Well, introducing lots of new APIs has its own problems of course.

It seems to me that it typically has fewer problems.
>Hearing about difficulties changes cause is good though (the reason for 
>alphas I guess) and if it is possible to work out why things are broken 
>then maybe we can still have the new functionality without the 
>breakage.

Of course.  Also, I should have pointed this out in my previous email, 
this information about failures is always easily available, at least for 
Twisted (and at most for any project interested in participating in the 
project), on the community buildbots page:

  http://www.python.org/dev/buildbot/community/trunk/

So anyone who cares to can check to see if their changes have broken 
things right away, instead of only finding out 6 or 12 or 18 months 
later. :)
>The problem with Lennart's report is that it is just "things are 
>broken" without much clue. I am sympathetic to this (working out 
>*exactly* what is broken in someone else's code can be a painful chore) 
>but I'm not sure what can be done about it without more detail.

Certainly.  Perhaps Zope would like to add something to the community 
builders page.

Thanks,

Jean-Paul