[IPython-dev] Notes from the testing trenches...

[Fernando Perez]

Hi all,

I'm very sorry that I've completely neglected all the (critically
important) discussion here over the last few days.  Basically I
decided that I needed to focus 100% on pushing on the testing work,
which I've become convinced was endangering seriously the long-term
viability of IPython.

I'm *very* happy to report this now:

tlon[plugin]> nosetests -vvs --with-ipdoctest --doctest-tests
--doctest-extension=txt IPython
... lots of output ...
Doctest: IPython.testing.tutils.fullPath ... ok

----------------------------------------------------------------------
Ran 445 tests in 75.199s

OK

Yes!!!  (the above includes all the IPython1 tests, in addition to
many scattered docstrings and small tests from IPython0, many with
actual IPython syntax in the docstrings).

This is the first time that we've had real testing for all of IPython,
and this has proved to be surprisingly difficult to get working.  I
have nobody but myself to blame for some of it, since precisely the
lack of clean tests led to an over-coupled API, but part of the
difficulty also stems from the fact that IPython, nose and doctest are
effectively all colliding and fighting each other for control of the
execution namespace.

In fact, there are still things that don't quite work right, and at
least one of them (having to do with namespace control) is very
important.  I have a solution in my head for it, but I need to sleep
now and it may be a couple of days before I get to it.

For now I'm putting all of this in a branch:

https://code.launchpad.net/~ipython/ipython/test-tools

where I've been regularly pushing.  I don't expect any of this to run
on anything but *nix (maybe not even OSX, I'm not sure) for now.
We'll eventually make it run under all platforms, but we're not there
yet.  If you're brave, you can pull that branch and go to the
IPython/testing/plugin directory, and type

make all

let me know how much blows up :)

I will be offline for a day or two (visitors), and will try to get
back to the notification/API discussion on Monday.  A big apology to
Barry and Gael for ignoring you, really.  But I'd neglected the
testing machinery on ipython for years, it was really time to fix this
or else it was going to drag the whole project down into the ground.

So if anyone is brave/foolish enough to try to run this branch and
these tests, please let me know what happens.  Eventually we obviously
need this to work for everyone, and once the machinery is stable it
will allow us to have much more strict policies regarding tests for
the project.

Cheers,

f