[Edu-sig] Migrating to Projects - Was: Low Enrollments - programming as anti-intellectualism

[Peter Hansen]

Laura Creighton wrote:
> (cc'd to Peter Hansen, because it is of interest and also mentions him - lac)

(Thanks Laura... I've temporarily subscribed to edu-sig in case my 
comments/opinions can be of value to someone.  If that concerns you, 
don't worry, I won't stay long! <wink>)

> In a message of Fri, 04 Nov 2005 12:31:03 MST, Trent Oliphant writes:
>>This brings up an interesting discussion point, one in which I would be 
>>extremely interested personally:  How do you help students (or yourself) 
>>move from writing scripts, functions, classes, modules etc. to writing 
 >>a larger project.
[snip]
>>I know it may seem that I am talking about an IDE - but even those 
 >>assume that you already know how to do a project.  So I get 
overwhelmed.
 >>I want them to work for me - but they just haven't yet.
[snip]
>>I am a lone programmer - entirely self taught - and this is a daunting 
>>task.  I feel confident that I could handle programming most of the 
>>individual components that would make up the project, but I don't even 
 >>know where to start and there appears to be nothing out there to help
 >>me learn that.
>>
>>Trent Oliphant
> 
> I think you are looking for the wrong sort of magic bullet.   What you
> need to learn is how to think about large projects.  

I could keep all of Laura's comments as they are, of course, right on 
the mark.  I'll comment directly on a couple of her points below and 
then say one or two things that come to mind in response to Trent's 
issues above.  I suspect this could be a long response so I'll write 
fast to make it easier on you. ;-)

> Whatever you get, make sure it is about Test Driven Design.  There are
> other competing methodologies for designing large programs, which have
> a long history, of which the most famous is called Waterfall.  All
> you need to know is to avoid them like the plague.

Other than to have a basic understanding of what those other 
methodologies are, and more importantly why they fail (often, and 
gravely), I'd agree that you can get pretty far with TDD alone in 
building large projects.

I think the first thing to understand is that successful large projects 
are just small projects that have grown.  It is possible, more so in 
theory than in practice, to "start big" by designing everything in 
advance, documenting the heck out of it in requirements specs and design 
docs and test plans, building the beast, testing it thoroughly, and 
shipping it.  Possible, but rarer than a white crow.

There are dozens of reasons why this is so, but I'll focus on the 
positive side here instead: how do you grow a small project?  You can 
write one module, then another, and so on, but you'll just rightly say 
"but I don't even know where to start."  And assorted random modules 
don't form a cohesive whole, and really this approach works only if you 
already did "big design up front" and know where you're headed.  Most of 
the time, nobody really knows that, even though many people delude 
themselves into believing that they do.

You could try writing one piece of functionality instead, even if it 
involves code in a few different modules.  Think of it as a "slice" of 
the future application, maybe a bit of GUI, maybe a bit of the 
algorithms, some persistence.  You'll quickly get into trouble here too, 
however, since when you start to evolve you'll have massive breakage, 
awkward interfaces, and no idea where you stand after only a few changes.

Test-Driven Development will help with this, though I don't personally 
find it sufficient.  What it will do (as you try this "slice" approach) 
is _lead_ you in the right direction as you code.  You are writing the 
tests _first_, before each chunk of code that will pass those tests. 
You are building a body of tests which get re-run often, ensuring you 
cannot write more than a few bad lines of code before an alarm goes off 
telling you you're going off track.  It forces you to implement less 
awkward interfaces, and gives you a measuring stick to tell you where 
you stand.

What I find it doesn't do is lead you from at a "big picture" level. 
What "slice" do you work on first?  How do you prioritize the other 
slices?  How complete do you make each feature in the first pass?  When 
are you "done"?

These are areas where complete "methodologies" come in, and where 
Extreme Programming has value for me.  Without a "methodology" (one 
could just call them project management strategies), you may have 
excellent, working code, maybe even lots of it, but you'll look back on 
it after six months and realize you've wasted the last four months on 
features non-essential to your users, maybe things even you don't value.

Trent says, "I'm a lone programmer - entirely self taught - and this is 
a daunting task.  I feel confident that I could handle programming most 
of the individual components that would make up the project, but I don't 
even know where to start ...."

I feel exactly like that at the start of every project!  (And, by the 
way, I'm self-taught in programming too, so I remember something of what 
you're feeling here.)

Really, even right now I'm responding to this email because I'm daunted 
by the two projects which I should be working on instead, one of which 
is really too large for one person, and the other of which, while I can 
handle, has discouraged me after we made a false start (customer 
specified wrong requirements, we wasted our time) and I haven't rebuilt 
my "gumption" enough to try it a second time.

Interesting side point to note, perhaps: these are issues which 
traditional "heavy-weight" processes like Waterfall pretty much ignore. 
  Human concerns, emotions?  Irrelevant.  This is _programming_. 
Technology!  Get your ego out of the way and use this tool correctly and 
you'll get the job done in no time.

Agile processes do deal with soft issues, however.  (See 
http://www.agilealliance.org for a start, and make sure to read the 
brief Manifesto before spending any more time on that site... this stuff 
doesn't "fit" some people's brains.)

In this case, XP helps me figure out small, achievable, prioritized 
steps.  It basically points me (with relatively little effort on my 
part) at where I should start, specifically, what precise "slice" is of 
most value _right now_.  It lets me define better how much of that slice 
I need, and thus defines the tests which I should write when I get into 
the actual coding (i.e. Test-Driven Development mode).  It let's me get 
going in spite of the daunting feeling (which I suspect never goes away).

I expect some folks here and certainly many elsewhere would disagree 
with much of this.  They feel that non-agile approaches work just fine, 
and in their own cases they might be right.  I can't argue about that, 
and I don't.  I do consistently come in to my new customers' sites and 
hear the same kind of talk.  "Look how we've got this nice design 
document we wrote late last year.  This is our master plan, our guide." 
  Universally these customers are _way_ off that plan, in most cases not 
even remotely doing what it says they should do, except in a very 
hand-waving kind of way.

They've talked themselves into believing that because they wrote a 
document which has a table of contents that appears to cover the entire 
scope of the system, they must be doing things right.  I'm skeptical, 
but have an open mind.  Maybe this is going to be the first customer 
I've had who really and truly can do things this way (though, if so, why 
did they call me in?  Hmmm...)

Then we get into the code, and start looking at the upcoming features, 
and I'm flipping through the design document looking for the matching 
text and finding empty sections with "TBD" (To Be Determined) written 
just under the titles.  A block diagram that has about six different 
major components that don't actually exist in the hardware.  A proposed 
schedule that ended five months earlier even though the code I'm looking 
at is barely prototype quality.  Signs that only two people in the 
company have a current copy, and one of those is missing the second half.

I could go on.  Thankfully I won't, and don't need to, because so far 
each of these customers is slowly and quietly letting go of their old 
ways as we come in and, often within a couple of weeks, start giving 
them a warm fuzzy feeling that their project, while late relative to 
their original expectations, and struggling with quality issues, is in 
the right hands.  Why?  Because we've identified some priorities, we've 
started writing some tests, and we aren't making unrealistic promises 
and more.  (And their pain is fresh, so they're meek enough for a while 
that we can push them around a little. ;-)

To build large systems, start small.  Break the work into small tasks, 
prioritize, work on them one at a time.  Write automated tests for 
everything.  Aggressively limit complexity, including in the code, the 
tool set, the technologies you are using, and the administrative 
processes with which your bosses smother you.  Use a revision control 
system.  Consider a wiki (for team work) and an issue tracker, but never 
add a tool until you've struggled without it for a while and are certain 
it will make your life easier.  Measure your progress and always adapt; 
go back and improve your approach the next time.

The following sites might be of interest, though unfortunately I do 
_not_ maintain a current set of useful links in the area.  I tend to do 
that for my own purposes as I'm learning, but once I've adopted a 
technique/tool/mindset I tend to stop reading about it and just use it.

(In this case, I've had this mindset for five years, which is the 
longest time any one approach has held my attention and give me 
benefits, and so far this one shows no signs of stopping.  But I don't 
believe it will always be my primary approach.)

  * http://www.agilealliance.org/ (mentioned above)
  * http://c2.com/cgi/wiki?AgileProcesses
  * http://www.extremeprogramming.org/ (somewhat dated)
  * http://www.xprogramming.com/
  * http://c2.com/cgi/wiki?TestDrivenDevelopment

I hope some of these thoughts prove helpful...

-Peter