application development - separating content from function

[Peter Hansen]

Henry Baumgartl wrote:
> 
> For eight months now, I am preparing for the development of an open source
> Python application to support individuals in Life Long Learning. It will be
> a mixture of PIM, extended diary, groupware, project management, and modules
> to interact with centralised knowledge / skill exchange servers.
[...]
> What needs to be done is:  [snip description of lots of work]
> 
> The more I have tried to solve these requirements in an elegant and
> efficient manner, the further I seem to stray from anything remotely
> resembling a viable solution. In other words, I can't see the forest for the
> trees anymore.

Welcome to the club! :-)  Undertaking an overwhelming project is pretty
common, especially with many open source projects.  Check out the number
of idle projects on SourceForge, for example.  I've made many many 
false starts in my life as a programmer, and I still regret much of the
time I spent on some of those hopeless projects.

My suggestion to you would be to investigate the practices and principles
of Extreme Programming, especially YAGNI (you aren't gonna need it), 
DTSTTCPW (do the simplest thing that could possibly work), frequent
releases and small iterations, using automated tests to drive the 
development and let the design emerge on its own, and perhaps most
important of all, given your situation, controlling scope.

You've got a really, /really/ huge pile of work ahead of you if you
think of all those pieces as being essential to the project.
Prune it down.  Think of the single most useful piece and exclude 
everything else.  Now think of the single most useful feature of 
that piece, and think of the single most useful aspect of that 
feature.... drill down until you have something that you think you 
can implement in a few days or a week of work.  Now do it.  Have 
automated acceptance tests for it so you know when you've got it 
done, so you can stop working. Release it to your users, or 
yourself.  Use it.  Get feedback from it.

Now... sit back and think about the next most useful feature that
you could implement.  Add it.  Make sure you have those automated
tests.  They let you be confident that the previous code still
works, that the new code works and that you're done when you're
done.  They let you aggressively modify the structure without
having to worry that you'll break something (you will, but you'll
find out immediately, back out the change, and carry on without
getting bogged down in lengthy debugging sessions).

If you picture your project as a very large number of potential
features being implemented one by one, you can probably think of
some that are obvious "must-haves", with a big bang for the buck.
There are others that maybe could be postponed a while, and probably
a bunch that you might never need to do.  Your goal is to pick
those items with the highest value first and implement them, 
without worrying about the others.  That way, if you have to
stop, you at least have something useful.  If you don't focus on 
doing the highest value items first, you'll probably find that
at some point you're ready to quit, you have nothing that can 
actually be used for anything, and you've spent a year and a
half of your life with nothing to show for it but a bunch of 
code.  (And in most projects, code without good tests, which is
as good as files filled with random bytes.)  With good tests, you
can always hand over the project to another.

I've found Python to be an incredibly effective language to use
in this kind of environment, especially as it makes the automated
testing as close to trivial as it probably can be.  But more than
Python itself, starting to think in XP terms has improved my ability
to produce useful, reliable code in a timely fashion.  If you
visit http://www.extremeprogramming.org/ for a quick intro, or
http://c2.com/cgi/wiki/?ExtremeProgrammingRoadmap for a much
larger set of info, you might gain a least a little inspiration,
even if you decide to stick with more traditional approaches.

-Peter