Python for large projects

[Hung Jung Lu]

> On Tue, 2004-03-23 at 17:24, Cameron Laird wrote:
>  
> > they're at a particular DISadvantage there.  If you have a
> > big job, you *particularly* need to look at Python (or Erlang,
> > or Eiffel, or ...)
--------------------------------------------
gabor <gabor at z10n.net> wrote in message news:<mailman.300.1080071082.742.python-list at python.org>...
> ...
> i wanted to use python for a project in our company... we wanted to
> build a fairly big system/program.
> 
> but when i recommended python, i got a question like:
> (previously all the programs were written in java)
> "if one of our programmers changes a method in a class/interface, we
> immediately will know about it, because the next program-rebuild will
> simply fail. but if we would use python, we wouldn't find it out".
--------------------------------------------

I use C++ and Python everyday. Let us be fair and point out some good
things about each of them.

(a) In compiled language like C++, changing function prototypes and
variable names is comfortable, because the compiler will find all
those spots that you need to change. In Python, you do not have the
same level of comfort. Sure, there are other techniques, but it's
different than clicking a button.

(b) Cameron said something very true in my opinion: for large
projects, you want Python. But he said so without giving more details.
So let me add some comments.

In my opinion, the essence of software development is code/task
factorization. It seems such a trivial concept, but if you really
really think about it, goto statements, loops, functions, classes,
arrays, pointers, OOP, macros/templates, metaprogramming, AOP,
databases, etc, just about every single technique in programming has
its base in the concept of code/task factorization. Take for instance
classes and inheritance, basically, you factor out the common parts of
two classes and push it up into a common parent class. To go one level
deeper, my belief is that at the bottom, all human intellectual
activities are based on factorization: no more, no less.

In large projects, you'll find that you need to factor out even more.
Let us take an example. Suppose you write an application, and later on
you realize that you need to make it transactional: that is, if some
exceptions happen, you want to roll back the changes. Now, this kind
of major after-thought is terrible for languages without
metaprogramming capabilities. To add a new feature, you will have to
make modifications in hundreds or thousands of spots. Another example,
suppose your software is versioned, more over, you have different
versions for the application and for the data file format, and your
application needs to work with legacy file formats. Again, without
metaprogramming capabilities, your code will have many redundant lines
of code, or be cluttered with tons of if-statements or
switch-statements. Another similar problem: you have several different
clients that buy your application, and they want some different extra
features. Again, without metaprogramming, your code will be either
hard to code (using virtual functions, function pointers, and/or
templates in C++), or will be cluttered with if-else- and switch-
statements (a terrible practice that will make your code
unmaintainable.)

As your project grows more and more complex (become threaded, many new
clients requirements, support for legacy versions, using distributed
computing in a cluster, etc.) you will realize more and more that you
need to factorize efficiently, otherwise your pain will be unbearable.

When you have reached that point, you'll come to appreciate simplicity
and purity in a language. Frankly, Python is good but still not good
enough.

For large projects, if you use a rigid language, then your best bet is
to use tons of programmers coding trivial interfaces and APIs to make
up for the shortcomings of the language. In flexible languages like
Python, you often can use metaprogramming features to factor out the
common areas. At that point, I think that issues like automatically
finding name changes as I mentioned in point (a) become small issues,
because you will have bigger concerns. The fact that you may miss a
name change or function header change is not the thing that will kill
you. The fact that your entire system is unmaintainable is the thing
that will kill you. Don't look at individual bugs when you are talking
about large projects, because your worry should not be there: your
worry should be focused on how to make your system maintainable. Bugs
can and will be fixed. But if your language does not allow you to
factorize efficiently, at the end of the day, that's what's going to
kill you.

regards,

Hung Jung