[IPython-dev] Feature and scope creep in IPython

[Nicholas Bollweg]

I started reading the thread back in december answering the perceived
question "i'm not core: should I keep helping?" and started writing code as
this is related to some of the research in which i am involved, and needed
something to do. :)

I have used a lot of code/effort management tools (open source and
otherwise), and pretty much none of their project concepts really scale in
all the dimensions that IPython is growing: codebase, distributed users,
platforms, security concerns, management (not in the bad sense),
modularity. How can the dense graph of information related to the broader
effort be parsed, quickly and visually, to find and answer the core
questions that are afflicting the project, such as the ones matthias
mentioned, specifically for pull requests:
 * This pr has 3 "merge ok", I just have to read add mine and I can merge.
 * This PR has already 2 dev with are "involved" I can worry of others.
 * This one has nobody I'll look after it.

Further, why should the choice of tools for one aspect of the community
(such as repos: monolithic vs modular, git vs hg, etc.) dominate the
interaction the developers and leaders experience (such as issues)? Why
should the data model of an upstream data vendor dictate what the users can
do, such as the inability for non-collabs to label tickets, when a beeelion
sites exist for doing tagging?

Here is the result (with the earliest bits of IPython's meta loaded)...
https://fra.gl/?/config/ipython
a lot remains to be done, but i think i can answer the latter two, I still
need to do some thinking about the former: you do have to log in to github
over oauth, and then configure the columns on the tables to start seeing
what i'm talking about. working on better doc/screencast!

Here is the code (bootstrap+d3... flask for OAuth. fabric for automation):
https://github.com/bollwyvl/fragile

metameta:
https://fra.gl/?/config/fragile

I was hoping to get it a bit further, but since this has come back up, I
thought I'd do the reveal, and maybe spark some interest.

I am also interested in taking this work and doing some experimental stuff,
hopefully to be written up as a(n) paper(s). If you're interested, check
out:
https://github.com/bollwyvl/fragile-papers

But the end goal is to help make transitions like the kind that IPython is
experiencing less painful.


On Thu, Jan 10, 2013 at 8:38 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> Hi folks,
>
> On Wed, Jan 9, 2013 at 11:12 AM, Brian Granger <ellisonbg at gmail.com>
> wrote:
> > I think this issue is really important for the project to consider if
> > we are going to successfully scale our efforts.  Please read this,
> > think about it, and post your thoughts here, as it related to IPython.
> >  At a practical level, we need to figure out better work flow for our
> > GitHub issues, especially ones for new features.  I propose one option
> > in my blog post, but I am sure there are others.
>
> I just wanted to mention that Brian has encapsulated here very much my
> thoughts as well (and not by accident, as we've been talking a lot
> about this).  It's important to note that this is NOT a 'hostile'
> attitude towards new contributions, since we are keenly aware that the
> value of IPython is in its utility to end users, who tend to always
> want new features for their problems.  And who are kind and energetic
> enough to often go out and contribute them, for which we're incredibly
> grateful.
>
> But this is a careful consideration of the painful reality that it's
> possible for a project to die under the weight of too much code, too
> many features.  So the challenge we are facing is how to keep a
> project that is very, very useful to people and allows them to do
> novel things, and yet stays tight and solid enough to be
> comprehensible and manageable by a relatively small team.
>
> I want to illustrate the kind of thing I see as where we want to go,
> with some examples from our own development:
>
> - the cell magics work: it enabled a ton of new features, but the
> underlying implementation was actually a major *reduction* of internal
> complexity, as we were able to factor a bunch of scattered
> functionality into a single model.
>
> - the merge of the parallel and interactive kernel into a single
> object: again, less code to be maintained but the automatic gain of
> all 'real ipython' functionality (magics, special syntax, etc) in all
> parallel contexts.
>
> - the ongoing effort to rationalize our input transformation machinery
> (IPEP 2 and PR #2447): while not done yet, we're trying precisely to
> get all the functionality we want with an ultimately simpler system.
>
> So we hope that it's clear that the vision carefully detailed by Brian
> is not a user-hostile position, quite the opposite.  It's simply
> driven by a very strong desire to ensure that, two years after we have
> put a solid team working full time on this, we are standing in front
> of a project that is tight, comprehensible and a very robust
> foundation for the work of others, and not a sprawling mess of many
> features that nobody understands internally.
>
> It is very easy to die under the weight of complexity, and we really,
> really want to avoid that.
>
> A big thanks to Brian for taking the time to articulate these ideas so
> well; one of the things we'll be doing more as we move forward (and
> Aaron pointed that out in the comments) is trying to lay out these
> ideas and vision in an explicit way, so that everyone has a common
> reference point they can go to, rather than working off implicit
> assumptions and finding out after a lot of work they had gone in the
> wrong direction.
>
> Cheers,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130114/4de517b4/attachment.html>