[IPython-dev] Status of pre-0.10 reviews and merges

[Fernando Perez]

On Thu, Apr 2, 2009 at 8:04 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:

> The big things we need to look at are:
>
> * If ipdoctest is picking up things that should really be run as tests
> that are somehow using twisted, but not in an officially "good" way
> * If there are globals that both twisted.trial and ipdoctest is
> mucking with that would be a problem.
> * The reactor is being started twice.  That is very odd, but could
> help us debug this.
>
> I think this will take both of us to tackle.  My idea is to start
> disabling parts of ipdoctest, or run the test suite on small parts of
> the codebase to try to isolate where the problem is coming from.

After all the time I spent on this one, I think we should  for now
leave it, since we have an ugly but functional workaround.  Given how
little explicit coupling there is between the two things, I suspect
that this will require making a specific branch (to be thrown away
when done) where we start deleting *tests* until we find which one is
causing this.  I suspect that it's not the one that raises the error
but something else up the chain instead.


>> I know we've talked a bit about this, but before we get started on
>> this major reorganization, let's make sure we do plan it carefully.
>> For better or worse, while ipython is formally 'pre 1.0' and as such,
>> all API changes are in principle fair game, we need to ensure we find
>> a good balance of practicality vs purity.  Just like numpy kept some
>> Numeric-inspired baggage that wasn't ideal so that users could
>> transition without unnecessary pain, we should keep the same ideas in
>> mind.  We should obviously fix the major messes we have (ipmaker is
>> ridiculous, 3 config systems is crazy, etc), but we should also be
>> willing to tolerate small warts if they make existing user's lives
>> easier.
>
> I am trying to distinguish between users and developer_users.  For
> users, I strongly agree that IPython shouldn't really change, or if it
> does, in a very limited manner that improves things drastically (like
> the config system).  For developers though, we need to allow much more
> freedom in changing APIs.  We won't be able to do the work without it.
>  But, even there, I agree that we should only change an API when we
> really need to.

Yup, I'm sure in the end we actually see things very similarly.

> But...,the first things I will be working on will be pretty different
> from the types of changes you are talking about here.  Basically, I am
> going to do a massive module/package reorganization.  A draft of the
> idea is this...
>
> I am going to reorganize the top-level modules into logical
> subpackages.  I am also going to use this as an opportunity to:
>
> * Identify rotting and broken code and move it to a directory (I won't
> delete it....yet) where everyone can look at the code and decide if we
> want to revive it (with tests+docs) or simply kill it for good.
>
> * Renaming modules and packages to pep 8 style lowercase.py
>
> * Adding tests to make sure that every moved/renamed module and
> package is importable (yeh, tests!!!) and working.
>
> * Where it is appropriate, begin to partition large modules (iplib,
> genutils, etc.) into smaller more logic submodules to make them easier
> to refactor and work with.
>
> * For really public facing developer APIs, I will put in a
> compatibility layer with deprecation warnings.

Big +1 to this.  By the way, in the disentagling front, keep in mind
my earlier comments about the biggest problems to fix being:

- ipmaker/iplib: ipmaker is kind of a 'super __init__' mess that needs
to disappear, folding it into iplib while making the lot be
rational/comprehensible.

- genutils

- making magics standalone

I hope these pointers help you a bit.

> The only thing that will change in the actual code though will be
> import statements and all of this will be tested.  No API changes will
> be combined with this reorganization work.  This reorganization is
> needed to help us really identify and isolate the things that need to
> be refactored moving forward
>
> But, because this work will move *everything* around, it will be
> difficult to merge branches back in in an automatic way.  By hand
> though it might not be too bad to merge things.
>
>> Part of that is communicating a plan for the breakage to everyone in
>> advance, so that feedback can be provided by all affected before
>> anything irreversible has happened.
>
> Yes, I will absolutely ask for feedback about these plans before
> starting.  I am working on an IPEP (IPython Enhancement Proposal) that
> describes my plans.  More details to follow soon...
>
>> Furthermore, I'd like to suggest that post 0.10 we hold a 'bug and
>> test weeks' (2-4 max) to see if we could do an 0.11 release where we
>> basically try to *only* fix existing tickets and add as many tests as
>> we can, before starting to do major API breakage.
>
> In theory this is a great idea.  In practice this timescale will cause
> problems with the work I need to do.  I currently have funding that
> ends May 1 to work on this and I need to get started immediately.  It
> is so rare to have dedicated time and $ for this, I don't want to
> waste the opportunity.  That is basically 4 weeks away.  In reality, I
> think I should soon just "go for it" and help people to do their
> merges into my branch by hand.

No problem.  Honestly we need this so badly, that it would be madness
from my part to get in the way.  So fire away :)


One suggestion: because LP only allows code reviews from logged in
users/ team members, let's have the bulk of this design discussion on
list rather than on LP.  I also like better having this archived on
our main mailing list for searching, etc.  Once we get into actual
code reviews for merging, we can leave that to happen on LP.

Cheers,

f