please comment on technologies

[Alex Martelli]

luc wastiaux wrote:

> In article <HqCqa.8931$3M4.241359 at news1.tin.it>, Alex Martelli wrote:
>>> The project would be developped by a group of 4 to 6 CS students
>>> (including me), over a three months timespan (but not full time, we have
>>> to go to classes and other stuff)
>> 
>> The most valuable thing you'll learn will be how to structure your
>> work.  I heartily recommend incremental development (most crucial
>> features first), test-driven design, pair programming, and a healthy
>> diffidence towards "big design up front" - you may get the temptation
>> of designing in some big chunks of infrastructure preparing for future
>> enhancements... that may never come.
> 
> In short, you recommend following extreme programming guidelines, did I
> understand your advice correctly ?

Not necessarily ALL of them.  "Customer on site and part of the team" may 
be particularly hard here for example -- if your "customer" is a professor
he may not WANT to be part of your team for undestandable reasons.  And
while "40 hours weeks" is a wonderful rule, it's hard to stick to when one
is a student in some engineering discipline -- what with the courseload
and all, I remember my time as a student as one of EIGHTY-hours weeks,
month in month out (looking back a quarter century, I can't believe I ALSO
managed to have something of a social life -- guess I must not have slept
for a few years on end;-).

But I'm definitely recommending some variation on *Agile* development
methodologies, and particularly recommending *AGAINST* the "waterfall"
model of development that you may have been taught and, in my opinion
and experience, is a disaster.  In "waterfall", you're supposed to do
all analysis first; then, all design; then, all the coding; and then
all the testing.  It just plain don't work.  It's better to plan for
incremental, iterative development from the start.  Do SOME analysis,
identify an absolutely minimal set of features you MUST have at all
costs, i.e., in your case, a set such that you'll fail your course if
you don't have AT LEAST those features.  Put all the "oh yeah and it
would be nice to have THIS too" on one side for now -- index cards do
work quite nicely for that (XP uses them, but so do many other approaches,
particularly agile ones).

Having identified an absolutely-minimal, will-die-without-these set of
features, then do SOME design -- as little as you can get away with:
i.e., design the minimal infrastructure you MUST have to implement the
features in the minimal set.  This is where some (at least preliminary)
technology choices MUST be made, too; for the purpose you'll probably
need "spikes", where two of you go off on the side for a bit (could be
hours or a couple of days) and try out some technological possibility
to ascertain it's well understood and stable and rich enough for you.

Avoid "big design up front" -- don't make the infrastructure rich and
powerful to support "future features" that may never come to be -- that's
an EASY trap to fall into.  What you DON'T want is to have the deadline
come and by that time have an analysis that supports 4 times as many
features as you could sensibly implement in the given time, a halfway-
done infrastructure that might support all of those 4-times-as-many
features plus as many again if they were ever dreamed up, and about 1/10
the working application code and about 1/100 the unit and acceptance
tests you should have.  I'm being light-hearted, but this IS what happens
in some software projects, particularly but not exclusively ones where the
participants lack actual in-the-trenches experience.

Divide the work up into SMALL increments, and make sure every one of
you is reasonably familiar with all the subsystems as well as the
overall architecture.  Pair programming, with ever-shifting pairs,
helps here.  One thing I've found helps things move forwards, even
though XP doesn't support it, is to have each of you take "primary
architectural responsibility" for some of the subsystems and modules
you have identified -- and don't forget to nominate one of you as
"overall architect".  This means that, while consensus is seeked, when
consensus cannot be reached the person who wears the architect hat
for the specific subsystem has to make the decision and get the whole
project moving again -- decisions can later be reversed, at the cost
of some rework, but I've found that trying to do EVERYthing by consensus
can slow things down excessively (a variant of "analysis paralysis").

*DON'T SKIMP ON TESTS*.  Once you THINK you have a design for some
subsystem, feature, module, class, function -- *WRITE TESTS FOR IT*
before you start coding it.  The tests will fail at first -- of course
they must, as the subsystem/etc ain't there yet.  But they DRIVE your
coding, acting as "executable docs and specifications" to some extent.

They're better than discursive docs/specs because they're executable --
thus necessarily unambiguous (not necessarily COMPLETE -- so, adding
SOME discursive material in terms of comments and docstrings is good
too -- but don't be surprised if NON-executable stuff fails to get
maintained and updated as your understanding grows... sadly, it almost
inevitably happens all of the time).

So -- I'm not specifically recommending XP for your needs (in some
areas, such as the advice to have "chief architects", my advice runs
counter to XP's) -- but I _AM_ recommending an Agile and deliberately
DESIGNED process for your work, AND avoidance of "waterfall".


>>> Spyce today and it looks like I could use it for the web interface.
>> 
>> Sure, that's one possibility.  Personally I'd choose Webware, which
>> offers an amount of flexibility I appreciate, but Spyce's OK too.
> 
> I will have a look a webware. I was looking forward to using spyce because
> it features <?php style tags to embed code into HTML templates, does
> webware have that ?

Webware does let you embed Python in HTML -- even though more often 
than not this is an inferior approach, Webware's designers are well
aware of the unhealthy fascination the approach holds for people
coming from PHP or ASP, and therefore support it.  But that's just
part of Webware's flexibility.  Servlets and templating are the ways
you should REALLY be using, IMHO.  Anyway, Webware supports them
all (AND lets you connect your code to the webserver with commendable
flexibility too -- so you can use one model when you're developing
and unit-testing and another for more extensive tests, for example,
without changing your code, just the "admin" procedures that you
use for starting it up).


>>> Then I have this idea about using XSL for templating, the idea is not
>>> only to separate the data from the representation, but also allow
>>> programmers to "leech off" the search engine (it's easier to do this if
>>> the output is XML).
>> 
>> The ability for other programmers to send "queries" to your search
>> engine and get XML output is good.  However, personally, I would avoid
>> XSL (unless you guys are already well familiar with it -- or NEED to
>> get very familiar with it anyway to pursue your studies).  It's simpler
>> to have separate URLs, or a form field requesting XML or HTML output,
>> and just use different templates with the same contents -- doing your
>> templating in Cheetah (or preppy/Spyce/whatever).
> 
> Ok, I thought XSL was the definitive answer to all templating needs, but
> you don't seem very excited about it ?

I guess I shouldn't put my name to this -- it's not a "respectable"
thing to say, and might inveigle me in flamewars I'd rather avoid -- 
but I think XSL is an approach that makes things harder for you, and
is therefore less pragmatically useful, than ad-hoc templating based
on mostly-procedural approaches.  It's worth learning it (particularly
if you have no other experience with non-procedural languages!) because
it expands your mind -- but I personally wouldn't want to have "mastery
of XSL" as a do-or-die prereq for any crucial project I'm responsible
for.  Perhaps I'm just not looking at it the right way.

> thanks a lot for all your comments and advice!

You're welcome!


Alex