[IPython-dev] iptest and the nasty twisted bug

[Brian Granger]

Ooops, didn't see this until now.

> I've spent hours trying to fix the interaction between twisted and
> ipdoctest, to no avail.  I have no clue what's going on, the problem
> is intermittent and makes no sense to me.  But what I realized is that
> at least us developers have a simple, if clunky, workaround so that we
> can run  the test suite by default in a twisted-robust way, while also
> running with full ipdoctest support.

We should talk at some point, I can probably help figure out what is going on.

> If you now run:
>
> iptest [any options] [any submodules]
>
> it will  NOT load ipdoctest but won't cause any Twisted problems.

That is good.

> Once you're happy that you didn't break Twisted, run
>
> iptest --with-ipdoctest [any options] [any submodules]
>
> This MAY give a Twisted AlreadyCalledError exception at the end, but
> it will also correctly load up all of the ipython-specific tests and
> doctests.

I think this is a pretty good compromise for now until we can really
solve the underlying problem.  We need to make sure that all the devs
understand this though.

My only hesitance is that I don't want devs to get used to see Twisted
related problems.  This makes it much more difficult to detect Twisted
related bug when they first show up.  Let's at least keep bug report
active on this.

> The above can be made easier with a trivial  shell  alias
>
> alias iptest2='iptest --with-ipdoctest'
>
>
> So that you can run
>
> iptest ...
> # Twisted happy
> iptest2 ...
> # ignore possible Twisted error, this checks all the rest.

> I know this is not ideal, but at least it does give us a way to run
> all the proper IPython tests and doctests.  I have not the foggiest
> idea how that error in Twisted is occurring: I'm not a Twisted expert,
> and the behavior is both intermittent and, to me, nonsensical.  The
> ipdoctest plugin in principle doesn't do anything with Twisted or any
> global state (its changes to global state are happening anyway even
> now, so that's not the problem).  I'm completely puzzled  by this.

Yep, these things are impossible to debug.  We will basically need to
find the problem by simply reading the code :-)

> **** But it is CRITICAL that all developers run the full suite with
> --with-ipdoctest ALWAYS before committing code  ****
>
> It may be annoying to have to run  the test suite twice, but that's
> the only way we can get test coverage of ipython magics and
> extensions.  In my branch there are already 90 more tests picked up
> this way, and this is only bound to grow as we add ipython-specific
> tests.  So these MUST be always run.

+1 on running it both ways.

> I'd love to find a cleaner solution,  and I'm sure we will eventually.
>  But for now, this is how it will need to be done...

Great, thanks for working through some of this!

Brian

> Cheers,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>