[IPython-dev] Test failures and errors in trunk

[Brian Granger]

======================================================================
>> ERROR: testExecuteFailuresEngineService_0
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>  File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/base.py",
>> line 729, in runUntilCurrent
>>    f(*a, **kw)
>>  File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py",
>> line 269, in errback
>>    self._startRunCallbacks(fail)
>>  File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py",
>> line 298, in _startRunCallbacks
>>    raise AlreadyCalledError
>> AlreadyCalledError
>
> This one, I've seen occasionally, and I remember we've talked about it
> several times.  It's completely non-deterministic, and you said that
> you'd also seen it some times.  I'd love to know how to track it, but
> I have no clue: it's obviously deep in Twisted.
> This isn't just a Heisenbug, it's a Twisted Heisenbug :)

Yes, we have seen this one before.  But a while back Barry and I
figured out where it was coming  from and we completely fixed it (so
we thought).  These are super difficult to track down.  If anyone see
odd Twisted bugs, please don't disregard them.  Usually, finding these
bugs means doing a line by line audit of all new code.  The problem is
that all the usual debugging tricks don't work (print statements,
debuggers) because of the non-deterministic nature of them.  But, I do
know what causes them.

I am going to back and find the last working revision of trunk that
lacks these errors and then start to move forward to see when the
problem appears.

======================================================================
>> FAIL: Doctest: IPython.testing.plugin.dtexample.ipfunc
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>  File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py",
>> line 382, in runTest
>>    raise self.failureException(self.format_failure(new.getvalue()))
>> AssertionError: Failed doctest test for IPython.testing.plugin.dtexample.ipfunc
>>  File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py",
>> line 27, in ipfunc
>>
>> ----------------------------------------------------------------------
>> File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py",
>> line 44, in IPython.testing.plugin.dtexample.ipfunc
>> Failed example:
>>    _ip.system("echo hello")
>> Exception raised:
>>    Traceback (most recent call last):
>>      File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py",
>> line 1212, in __run
>>        compileflags, 1) in test.globs
>>      File "<doctest IPython.testing.plugin.dtexample.ipfunc[3]>",
>> line 1, in <module>
>>        _ip.system("echo hello")
>>      File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py",
>> line 137, in xsys
>>        sys.stdout.write(commands.getoutput(cmd))
>>      File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py",
>> line 44, in getoutput
>>        return getstatusoutput(cmd)[1]
>>      File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py",
>> line 54, in getstatusoutput
>>        text = pipe.read()
>>    IOError: [Errno 4] Interrupted system call
>
> Now, these stump me: you see several of these, which I've never ever
> seen.  I wonder if the commands.py module behaves differently on
> different OSes.

> It's important that we can run these tests reliably, because those are
> the ones that let us do full 'real ipython' testing.

I agree that this is important.  But this does sound subtle.  One
possibility is that this is a side effect of the above Twisted bug.
But again, I think I will have to simple rewind to the last revision
that was clean, and then star to move forward through the merges.
This is just messy when there have been so many big merges.

>> Ran 515 tests in 57.113s
>> FAILED (SKIP=11, errors=3, failures=2)
>
> In all, you get three errors, so it's not *that bad*, considering we
> added over 100 new tests.  Those are:

True.  And like you say, the bright side is that we have a test suite
that finds bugs!!!!

> 1. The twisted heisenbug.  No idea, this one has been haunting us for
> well over a year (I remember talking to you about it back when I was
> still in Boulder).  It's only that now I see it more often (I'd say
> now I see it ~80% of the time, before it was more like ~40%).
>
> 2. The Numeric dependency.  Easy to fix.  In fact, I need to clean
> many more of those, for people who don't have MPL or other packages
> (c.f. conversation earlier with Jorgen).

Yes, this is trivial, but it is really important that we regularly run
the test suite on Python installs that don't have any of the
dependencies installed.  Much of the extra effort that arose in doing
the 0.9 release was catching these types of things.  I almost think we
need an API that allows modules or package to declare what they depend
on, so we can write tests to make sure our tests work (if that makes
sense).

> 3. The IOError on OSX.  I have no clue on that one, but I'll think
> about it.  It's extremely strange.

The first and third of these are in the "painful" category.  I guess
at some level it is my fault though for not running the test suite
after each merge is done.  Oh well.

> I'm actually much more worried about the fact that today, John Hunter
> discovered that my lovely fix to
>
>  https://bugs.launchpad.net/ipython/+bug/269966
>
> actually can cause problems with MPL (open figure objects hold
> references to the old module objects that get zeroed out).  I'm
> actually starting to wonder if this isn't a bug with Python itself, in
> how it clears module objects ignoring reference counts (or that's what
> appears to be happening, at least).

Sounds subtle.  You would never guess that a project like IPython
would have to deal with such subtle things.  It is pretty amazing.

> Relax :)  That's why I wanted all that testing support code merged in:
> because by enabling many more tests to run, it's likely to show us
> these problems.  Now we can actually target them for fixing :)

Yep, I definitely agree with this.  Sorry about my test failure grumpiness :)

Brian

> Cheers,
>
> f
>