Python evangelists unite!

[Peter Hansen]

Peter Milliken wrote:
> 
> "Peter Hansen" <peter at engcorp.com> wrote:
> >    ... we have had an order of magnitude
> >    fewer reported bugs, and have productivity at least two, probably
> >    three times, higher than on any project I've worked on in the past.
> >    These numbers are largely attributable to Python, and that makes
> >    using it good business sense no matter how you look at it.
> >
> 
> Good to hear your experiences. You look after 14 people - well done, that's
> a big team, do you manage to do any development yourself with that number of
> people to supervise? I never could with teams that size, it drove me to
> distraction......

Some.  Enough for now.  Not much.

> What is the basis for your comparisons though? You state that you have
> achieved fewer bugs and increased productivity using Python. (just for my
> information :-)) it would be nice to know over what other language you claim
> these improvements. 

C, C++, Java, Delphi as primary languages involved in the comparison.
The kind that emphasize (obviously not C :-) things like data hiding,
type checking, compile-time checking, and so on (the things too often
claimed as being required for writing "real" programs but lacking from
Python and therefore making it unsuitable).  Many others before and
along with these with no better results.  (I did learn Ada, but
not Ada95. :-)  (But after 100,000 hours of writing software or
thinking about it, I forget which languages I used to know.)

> Have you considered that there might be other languages
> available that provide similar order of magnitude improvements over Python?

Except in specialized areas, very, very doubtful.  I'll consider it 
seriously when I see the slightest evidence of such a thing.  I do
keep abreast of the industry and stay alert for more data.

> What was the basis for your choice of Python as the development language of
> choice? i.e did you perform any studies into available languages, or did
> someone recommend it? If it was recommended, what was the basis of the
> recommendation? Have you considered mixing languages i.e. Python for the
> bits that suit python and some other language for the rest?

We contrasted it with other languages with similar capabilities (fitting our
requirements at the time), including Perl, TCL, Ruby, others.  Not
recommended by anyone, except as discovered during research.  And yes, 
naturally we mix Python as needed.  We just rarely find the need, and more
often than not find that the benefits of sticking with Python outweigh
the minor apparent advantages of diluting our focus.  (For example,
C is used from time to time for performance or integration with
hardware.  Almost all other areas we now find more effective
to do with Python, increasing the gains from code reuse, training,
utilities, test frameworks, and so forth.  If we encounter a 
situation where another language will clearly be better we'll use
it.... but we're still waiting.)

> Having done some industrial control software when I was a
> "pup" what features of Python do you find useful here that you can't find in
> some other language? Does the interpreted nature of Python cause any speed
> difficulties? What is the required responsiveness of yoir problem domain?

Speed is not an issue.  As I've written elsewhere, I believe 95%
(or was it 98% last time? :-) of all optimization is an unnecessary
waste of programmer time.  We recently had to optimize a routine in
Python.  It's an unusual occurrence.  Responsiveness varies from
tens of microseconds (some embedded stuff for which we obviously
use straight C, not Python) up to several seconds, where Python is
more than adequate even on 100MHz 486.  Advantages of Python (not
necessarily missing from other languages, but perhaps not 
available in any single other one, and certainly not implemented
in the same, simple, effective way) include its introspection,
dynamic typing (yes, an advantage!), named arguments, incredible
readability, maintainability, and learnability, its interpreted
nature (interactive console lets developers test small portions
of code during development, so the result is already valid 
*before* the application is run the first time, and it saves
looking up APIs), cross-platform support (Windows and Linux),
very strong standard library, incredible community support, 
licensing (well down the list), capabilities as a "glue"
language (highly valued when we wrote the factory automation
application which talks GPIB, writes XML, has a GUI, multitasks,
talks to a serial port, talks to a CAN interface, wraps some
Windows DLLs, and does a half dozen other things), and 
probably a few more which I won't think of tonight. :)  Oh,
yeah, dictionaries, regular expressions, Zope, and more.
Oh, wxPython is great, too.  And it's great for writing 
automated testing (yes, a circular argument, since we rely
more on automated testing *because* it's Python.)

> > ...  Or perhaps the advice that a "poor workman should not
> > blame his tools" should apply even to those calling themselves software
> > engineers.  (If I were you, I would probably have put a smiley here.)
> >
> I'll put in as many as you want - I intend no offence in any of my comments,

Often people overuse smileys in postings with frequent sarcastic remarks,
apparently to let belittle others even while they claim not to have meant it.
I'll accept you didn't intend any of your comments that way!

> I am pursuing this thread in the hopes of (self) improvement :-). Lets see,
> information hiding, type checking, run-time checking, design by 'contract'
> (ties in with information hiding), spring to mind as immediate features that
> I would like to see in a language and that I was taught are "good features"
> to have. Seems you don't find them important, I guess we will have to
> disagree as to what is important or not :-)

Agreed. :-)  I used to think information hiding was important, and 
that type checking provided value to *me* (and not just to the
compiler).  Python's pragmatic approach (self._dontuseme) 
to the issue and the number of times it has been useful during
development or testing *not* to have to fight with the Java/C++ 
type of strictness has convinced me otherwise.  I assume you
mean compile-time checking above (?) (not "run-time checking") 
and PyChecker provides a large part of the same value.  The rest
should (I now believe) be caught by adequate testing, since
compile-time checking becomes a crutch (people end up relying
solely on it, and don't do the other tests).  No reason
you can't do design by contract, if you wish.  Test-first
design provides a more efficient way of handling the issue.

But the part about our training is interesting, too.  I was 
not taught much about information hiding and type-checking.
I was in an engineering program which emphasized systems
design, looking at complexity of systems, coupling and
cohesion, design methodologies, and so forth.  Little 
directly to do with software, as it tried to move to a
"higher" level (so they claimed) and focus on issues which
apply to engineering in any field.  As a result, perhaps
I'm less tied to (or trained in :-) these things which
I consider minor and mundane details of certain programming
languages, and look more at the benefits Python brings
to those other issues.  Code, for example, appears to
be significantly simpler in Python than in many other
languages.  The result is less complexity, easier
debugging and maintenance, faster development, and so on.
*These* are the engineering principles I keep in mind
and on which I judge languages.

> Personally I love information hiding and coding by 'contract' - it seems
> there is always someone in the team who is just plain lazy and wants to
> access and modify the behaviour of another object directly rather than using
> the agreed interface or requesting that the agreed interface be changed.

When (or if, really) this happens, we'll just refactor the
problem out of the code.  It hasn't happened yet.

> Strong typing is wonderful too. My Ada programs have 95%+ of *all* bugs out
> of them by the time I get it to compile cleanly - wished I could say the
> same of my Python programs! :-) 
> Finding bugs by testing is just so *expensive*!

Not if you write the tests first...  

> Do you gather metrics on the problems that PyChecker is meant to cure? i.e.
> does anyone record the fact that they lost 1/2 hr to the fact that they
> mispelt a variable name? :-) Metrics is the best way to improve your shop -
> unfortunately the typical programmer doesn't like to record them :-).

Our metrics are poor so far, but then again resources have
been exceptionally tight and we've done our best under the
circumstances.  But if problems caused by things like misspelt
variable names showed up much (or at all... not sure anyone's
actually been caught by it in the last year), we would focus more 
attention on the problem.  These things just haven't been
a problem for us, so we given them little thought.

> > So your suggestion doesn't seem likely to reveal any significant
> > deficiencies to me: if I haven't seen any myself, it must be because
> > *for my purposes* there are none!
> 
> Hmmm.... at the risk of offending, have you considered that you might not
> have sufficiently broad experience to be able to determine that?

I'm considering it.  Yes, of course that's possible.  No one
could say otherwise.  I just have no evidence to think that's
the case, and I believe I have quite enough experience (not 
quite your level, but twenty something years developing 
professionally, and probably 100,000 hours as I said above) 
to judge when I have a problem to solve.  The last time I 
judged that to be the case, I went looking for a solution.  
That's when I found Python. :-)

> > > You could stick with Python as it matures ...
> >
> > How much more than 10 years old does a language have to be for you
> > to call it mature?
> 
> 10 years is a long time, I guess from what I have observed in some posts on
> the list, people are attempting to state that they would like to see things
            ^^^^^^
Not "people", newbies. :-)  (Yes, I know they're not all newbies.)

> like some facility in the language to make it easy for them to type-check
> their arguments (one example that springs to mind). I really don't spend too
> much time looking for these items though or I might be able to pull up many
> more examples :-) Unfortunately, I "skim" these because I know there are
> very good languages that already offer these features (and they didn't take
> 10 years to get there! - which is *not* a slame at Guido - he has done an
> excellent job! :-)).

Don't kid yourself.  Python is not going to arrive suddenly at 
static type-checking after fifteen years and realize it was 
wrong all along.  It *might* end up with such a thing, probably
more to support compiler optimizations than anything, but some
of us (probably many of us) will be quite uninterested in it.
Guido has (in my opinion, as a supposedly well trained 
systems design engineer) an excellent design sense.  Some of
these things were left out for a reason.

> All either of us can do is talk from personal experience. I have a
> reasonable amount in that regards but it is limited in the sense of 29 years
> work experience in certain fields and situations - first 8 in industrial
> control, next 20 in defence (biggest project was 100+ programmers) and the
> last year in a "commercial" environment. So I  am trying to gather as much
> as I can from other people (we can't learn it all! :-)).

Ahh..  You are much more experienced than I am in really hardcore
heavy-metal development (or whatever euphemism I can apply to the
defence industry :-).  I seem to be much more experienced than you
in the commercial environment, where the issues can sometimes be
quite different (I believe).  I'm still working in the commercial
area, while you are freshly arrived, but I won't presume to think
that makes me more right than you.  

I *might*, however, have found that in this environment, some of 
the rigorous controls provided by other languages actually get in 
the way of solving the problem in a high quality and efficient 
manner (high quality being defined as satisfying the customer, 
and efficient being defined as before profitability vanishes :-).  
Actually, I believe that's exactly what I have found, and Python 
is a part of it.  (XP is another part, by the way, but that's a 
discussion for another newsgroup. :-)  Either way, I wouldn't be 
at all surprised if my conclusions would be all wrong in your 
former field.

-- 
----------------------
Peter Hansen, P.Eng.
peter at engcorp.com