Unification of Methods and Functions

[James Moughan]

David MacQuigg <dmq at gain.com> wrote in message news:<q313a0lroe1uiiu43nkhhnqnbcm4jhgkrk at 4ax.com>...
> On 10 May 2004 16:53:06 -0700, moughanj at tcd.ie (James Moughan) wrote:
>
> >David MacQuigg <dmq at gain.com> wrote in message news:<889t90tdl9o9t25cv5dj6k5rnktuce0jin at 4ax.com>...
> >> On 8 May 2004 07:07:09 -0700, moughanj at tcd.ie (James Moughan) wrote:
>
> < snip topics we have finished >
>
> >You are not solving a problem; that's the problem. :)  If there were a
> >real programming task then it would be more trivial to show why your
> >object model is broken.
>
> I could give you an example from IC Design, but for the course I
> teach, I chose to use a similar hierarchy based on something everyone
> would understand - a taxonomy of animals.  Nothing in this example is
> something you wouldn't find in a real program to model an integrated
> circuit.  Instead of animal names like Cat, we would have the names of
> cells in the hierarchy, names like bgref25a.  Instead of a variable to
> count the number of animals at each level, we might have several
> variables to track the total current on each of several supply lines.
> Like the counts in the Animals.py hierarchy, we need the total current
> to each cell, including all of its subcells.
>

As far as I understand it, this is fairly different; the way I would
implement it, each cell would be an instance, not a class.  I would
have to look at some kind of code/spec to really know, though.  Do you
have an example of a piece of software were you follow this approach?

> I'm sure there are other examples from other specialties.  In
> accounting, I can imagine a hierarchy of accounts, with a total for
> each account including all of its subaccounts.  Don't just assume that
> the problem isn't real because you haven't encountered it in your
> work.

Sorry, but that's not how you would do an accountancy package in OO. 
Again each account would be an instance, not a class, otherwise adding
a new account would require either code modification or metaclass
hacking.  Instances scope down, and having a count of the sub-accounts
stored in an object would be OK in my book.  Though you wouldn't need
to, since the container for the instances would do it for you.

But even this isn't how you would organize an accountancy program. 
You'd use a database.

>
> <snip>
>
> >If you can't take it below 70 pages and you only have 4 hours... maybe
> >it's not such a great idea to try this?  I can't see your students
> >benefiting from what you're proposing to do, if you have so little
> >time.
>
> I think I could do it in 30 pages and 4 hours total ( lecture, lab,
> and homework ), but not if I need to cover the topics that both Mark
> Lutz and I consider important to basic OOP in the current version of
> Python.  The 30 pages assumes the unification of methods and functions
> that I have proposed.
>

70 -> 30 because of this?  Really?

Anyway, I guess you know what you have time to do, it being your
course 'n all. :)

> <snip>
>
> >>
> >> What data are we talking about?  numMammals is specific to Mammal.
> >> genus is specific to Feline, but *inherited* by instances of a
> >> subclass like Cat.
> >
> >The numAnimals etc... data, which is stored in Animals but gets
> >arbitrarily altered by the actions of subclasses of Animal, and
> >therefore is not specific to animal; it doesn't represent the state of
> >the Animal class or of Animal objects, but of a whole bunch of
> >subclasses of Animal.
>
> The total current to an IC is the sum of the currents to all of its
> subcircuits.  That current is a single number, for example, 35
> microamps.  It has a name "Iss".  Iss is a characteristic of the IC
> which appears in data sheets, etc.  It is a variable representing the
> state of the entire IC.  It does not represent the state of any
> subcircuit in the IC, even though it gets "altered" whenever one of
> those subcircuit currents changes.
>

So the IC is an instance which stores or accesses data about the
instances which it contains, i.e. circuit elements; not a class which
you sub-class to get subcircuit elements.  I'm discussing classes and
class heirarchies, not instance heirarchies.

> Looks like this whole argument comes down to what we mean by the word
> "specific".  Let's drop it and focus on the more interesting topics in
> this thread.
>

>From here, it looks like the problem is the difference between an
instance and a class?


> >>
> >> These are normal programming errors that can occur in any program, no
> >> matter how well structured.  I don't see how the specific structure of
> >> Animals.py encourages these errors.
> >
> >Imagine if your structure had been implemented as one of the basic
> >structures of, say, Java.  That is, some static data in the Object
> >class stores state for all the subclasses of Object.  Now, someone
> >coming along and innocently creating a class can break Object -
> >meaning that may break anything with a dependency on Object, which is
> >the entire system.  So I write a nice GUI widget and bang! by some
> >bizzare twist it breaks my program somewhere else because of an error
> >in, say, the StringBuffer class.  This is analagous to what you are
> >implementing here.
>
> I'll need an example to see how these general worries can affect the
> Animals_2 hierarchy.  What I see is quite robust.  I added a Feline
> class between Mammal and Cat, and I had to change only two lines in
> the Cat class.  ( And I could avoid even that if I had used a "super"
> call instead of a direct call to the Mammal functions.)

And if someone unfamiliar with the code neglects to call the
superclass initializer when they create a new animal, then the code
will break in unpredictable places.  Not calling the superclass is a
common problem - but easily fixable, *providing* the effects show up
in the subclass, and not at some other random place in the heirarchy.

>
> >While errors are always going to happen, OOP calls on some conventions
> >to minimize them.  The most absolutely vital of these is that it's
> >clear what can break what.  Generally I should never be able to break
> >a subsystem by breaking it's wrapper; definitely I should never be
> >able to break a superclass by breaking it's subclass; and I
> >*certainly* shouldn't be able to break a part of the system by
> >changing something unconnected to it.  The whole of OOP derives, more
> >or less directly, from these principles.  Expressions like 'A is a
> >part/type of B' derive from this philosophy, not the other way around.
>
> Sounds good.
>
> >Your program breaks with this concept.  It allows an event in Cat to
> >affect data in Mammal and in Animal, which also has knock-on effects
> >for every other subclass of these.  Therefore it is bad object
> >oriented programming.
>
> We are modeling the real world here.  When you add a lion to a zoo,
> you add one to the count of all animals.

When you add a lion to a zoo, you add one entry in the lion table of
your database.  When you want to know the number of animals in the
zoo, the database query counts the number of entries in all of the
animal tables.  *Real* databases are built that way because of
experience; repetition of data invariably causes out-of-synch
problems.

I worked on a project a couple of years ago to port a legacy database
into MySQL, and it was made damned close to impossible by this sort of
thinking; how the heck do you port a database which has had it's
shortcomings patched over to the point where it depends on the
inconsistent results for it's output? :-\

> When you add 2 microamps to
> the core currents in a bandgap voltage reference, you add that same 2
> microamps to the total supply current.
>

Again, I'd be inclined to handle this problem at the instance level,
not the class level.

> I'm no expert in OOP, but what I have seen so far is not near as clear
> in structure as the origninal Animals_2 example.
>

When you have to troll through a 20K line program where the
functionality for a single subsystem is scattered through multiple
classes in a heirarchy, it puts a really different perspective on what
'clear structure' is.  Likewise, when everything is modularized
explicitly and you can alter a complex system which you barely
understand to include completely new core functionality in an hour or
two of work.

> >It takes us back to the days before even structured programming, when
> >no-one ever had any idea what the effects of altering or adding a
> >piece of code would be.
> >
> >It is therefore not a good teaching example. :)
>
> I'll need to see something better before I abandon the curent example.
> The problem may be our expectations of OOP.  I see classes as modeling
> the real world, including variables that are altered by changes in
> subclasses.  You seem to have some computer science notion of what a
> class should be.  I'm not saying its wrong, but unless it helps me
> solve my real-world problems, in a better way than what I am doing
> now, I won't use it.
>

You're right, I don't give a damn whether classes model the real
world; what matters, in general order of preference, is:

- Robustness
- Maintainability
- Programmer time
- Efficiency

I will sell any theoretical principle to the salt mines for those. 
'Classes-model-the-real-world' is an abstract theory, an evolutionary
holdover from when OO was the Next Big Thing in AI.  If it leads you
to design a non-robust, hard to maintain system then, in my book, it
gets dropped.

> I'm reminded of the criticism Linus Torvalds got when he first
> published Linux.  The academic community thought it was the worst,
> most fundamentally flawed design they had ever seen.  It did not fit
> some expectation they had that a "microkernel" architecture was the
> proper way to design an OS.  Luckily, Mr. Torvalds was not dependent
> on their approval, and had the confidence to move ahead.
>

I've been trying to explain why the pattern in your example will cause
bugs; you've been justifying it in terms of OOP metaphors. *shrug* The
comparison is clear enough to me... :)


> >> >> I'm not sure what you mean by "side effects" here.  The show()
> >> >> function at each level is completely independent of the show()
> >> >> function at another level.  >
> >> >
> >> >But the inventory data isn't independent.  It's affected by classes
> >> >somewhere else in the heirarchy.  Worse, it's done implicitly.
> >>
> >> The "inventory data" actually consists of independent pieces of data
> >> from each class. ( numCats is a piece of inventory data from the Cat
> >> class.)  I'm sorry I just can't follow this.
> >>
> >
> >numMammals OTOH is not just a piece of data from one class - it's a
> >piece of data stored in one class, but which stores data about events
> >in many different classes, all of which are outside it's scope.
>
> Exactly as we see in objects in the real world.

Objects perhaps, but not classes.  This seems to be the distinction
which is driving this whole problem; you are regarding extending a
class as if it were adding a member to an instance.

> >>
> >
> >That's the way it has to be, if you want to write it like that.
> >However there is nothing to say that a given problem must use a
> >certain class structure.  If you come up with a solution like this
> >then it's near-guaranteed that there was something badly wrong with
> >the way you modelled the domain.  Either the program shouldn't need to
> >know the number of instances which ever existed of subclasses of
> >mammal or else your class structure is wrong.
>
> Trust me, the need is real.  We just need to find the optimum example
> to show how Python solves the problem.
>

Mail me some code where you need to do this in a real system, and I'll
show you how to refactor it. :)

> In my work as a software product engineer, I've learned to deal with
> two very common criticisms.  1) The user doesn't need to do that.  2)
> The user is an idiot for not understanding our wonderful methodology.
> These are generally irrefutable arguments that can only be trumped by
> a customer with a big checkbook.
>
> I generally don't engage in these
> arguments, but on one occasion, I couldn't resist.  I was trying to
> show an expert how a complicated feature could be done much more
> easily with simpler functions we already had in our program.
>
> His argument was basically -- every expert in this company disagrees
> with you, and you're an idiot for not understanding how our new
> feature works.  I replied that I was the one who wrote the User Guide
> on that feature.  He started to say something, but it was only a
> fragment of a word, and it kind of fell on the table and died.  There
> was a few seconds of silence, while he tried to figure out if he could
> call me a liar.  I just looked right at him without blinking.
>
> Forget what you have learned in books.  Think of a real zoo.  Think
> how you would write the simplest possible program to do what Animals_2
> does -- keep track of all the different classes of animals, and
> display the characteristics of any animal or class, including
> characteristics that are shared by all animals in a larger grouping.

What you're asking is how would I implement something simply,
providing that I have to implement it with method X.  Give me a
functional spec - what the system needs to do from the user point of
view - and I'll tell you how I would do it.

In the case of a real zoo, I can pretty much guarantee that it would
begin

>>>import PySQLdb

>
> >And, as general rule, you should think carefully before using classes
> >to store data; that's typically what objects are for.  I used static
> >data in programs quite a lot before I realised that it too-often bit
> >me later on.
>
> Classes *are* objects.

In Python, classes are 'first class objects' - you can pass them
around as references, alter them and so on.  That's in the same way as
functions are 'first class objects'.  That's a different and older
piece of terminology than the term 'object' in OOP, which unhappily
coincides with it.

Wherever I can, I'll convert the discussion over to use the term
instance if you're more comfortable with the terminology, but in the
canonical definition a class is not an object.  Objects get created
from classes, and the term is equivalent to instance for practical
purposes (though you get into occasional pieces of terminology like
'object instance' where it's just convenient to not have to say
'object object', and 'object variable' would get kinda confusing too.)
 Check any basic text on OOP for confirmation, say
http://java.sun.com/docs/books/tutorial/java/concepts/


> I think you mean instances.  I make a
> distinction between class variables and instance variables, depending
> on whether the variable is different from one instance to another.
> Every instance has a different cat.name, but all cats share the genus
> "feline".  In fact, they share that genus with all other members of
> the Feline class.  That is why I moved it from Cat to Feline as soon
> as our example was big enough to include a Feline class.
>
> >
> >OK: start with the basics.  We need iterative counting data about the
> >individual elements of the heirarchy.
> >
> >The first thing is that we need to factor out the print statements.
> >Your back-end data manipulation modules should never have UI elements
> >in them.  So, whatever form the data manipulation comes in, it should
> >be abstract.
>
> You are adding requirements to what I already have.  OK if it doesn't
> slow the introductory presentation too much.
>
> >Secondly, we want to keep the data stored in each class local to that
> >class.  So, Mammal can store the number of Mammals, if that turns out
> >to be a good solution, but not the number of it's subclasses.  OTOH we
> >could remove the data from the classes altogether.
>
> Think of a real zoo.  If you ask the zookeeper how many animals he
> has, will he tell you only the number that are animals, but are not
> also lions or tigers or any other species?  That number would be zero.
>

In a real zoo, it would not require brain-surgery on the zookeeper to
introduce a new type of animal. :)

Also, as I say, in a real zoo you would use a database for this task. 
Any good database design person would balk at the idea of even storing
the number of a particular animal as an explicit variable in the
database.  Think about that.

> I really do want numMammals to display the total number of all
> mammals, whether or not they are a member of some other class in
> addition to Mammal.
>
> If I were to guess at your objection to this, I would assume you are
> worried that the different counters will get "out-of-sync", if for
> example, someone directly changes one of these variables, rather than
> calling the appropriate functions to make a synchronized change.
>
> My answer to that is to make the counter variables private.  I've
> added a leading underscore to those names.  numMammals is now
> _numMammals.
>

OOS due to an explicit variable alteration is pretty much the
worst-case scenario, yes.  It would require quite an incompetent coder
to do it, but it could still easily happen.  The more likely problem
is the failure to call the superclass constructor - which may even be
harder to debug, since at least in the other case you can run through
the program in a debugger and find where the variable is being
changed, or just grep numMammals.

Your faith in private variables is touching. :)

> >Thirdly, it would probably be nice if we had the ability to implement
> >the whole thing in multiple independant systems.  Currently the design
> >only allows one of "whatever-we're-doing" at a time, which is almost
> >certainly bad.
>
> ???

Supposing I want to store data for two zoos.  If the data on the
number of animals is stored at class-level, then I literally can't do
that without closing down the program and opening up a different data
set.  Munging the code to have numMammals_at_zoo_1 isn't a good
solution either. :)

>
> >After a bit of brainstorming this is what I came up with.  It's not a
> >specific solution to your problem; instead it's a general one.  The
> >following class may be sub-classed and an entire class-heirarchy can
> >be placed inside it.  It will then generate automatically the code to
> >keep a track of and count the elements of the class heirarchy,
> >returning the data you want at a method call.
> >
> >This is done with a standard OO tool, the Decorator pattern, but
> >ramped up with the awesome power of the Python class system. :)
>
> My non-CIS students are not familiar with the Decorator pattern.  I
> fear that will make this example incomprehesible to them.
>
> >
> >The above code is 51 lines with about 10 lines of comments.  For a
> >project of any size, this is a heck of an investment; I believe it
> >would take a fairly determined idiot to break the system, and *most
> >importantly*, they would be able to trace back the cause from the
> >effect fairly easily.
>
> This is an impressive bit of coding, but I can assure you, as an
> introduction to OOP, it will blow away any non-CIS student.  It may
> also be difficult to modify, for example, if we want to do what
> Animals_2 does, and provide a custom display of characteristics for
> each class.
>

Hmm, do you mean the talk method or the formatting of the inventory
data?  Because of the rather convoluted scoping rules you have to call
say animal_farm.<superclass name>.talk(self) to access the superclass
from within the talk method, which is a minor annoyance I guess.

To format the show method differently for each class would be a little
more awkward, though you're not doing that at the moment.

> One possibility is to make this an Animals_3 example.  Animals_1 was a
> simple two-class structure.  It served to introduce instance
> variables, and some basic concepts like inheritance.  When we moved to
> Animals_2, we pointed out the limitations of Animals_1, like not
> having enough classes to put variables like 'genus' where they really
> belong.
>
> Maybe we should go one more step, and make this a third example.  We
> can point out the limitations of Animals_2 in the introduction to
> Animals_3.  I can see the benefit of moving the print statements to
> the top level.  This is needed if we ever want to make the classes in
> Animals_2 work in some kind of framework with other classes.  The
> show() functions in Animals_2 could be modified to return a list of
> strings instead of printing directly to the console.
>
> I've posted your program as Solution 3 to the exercise at
> http://ece.arizona.edu/~edatools/Python/Exercises/  Could you give us
> a brief description of the advantages and disadvantages compared to
> the original.  I'm not able to do that, because I'm having difficulty
> restating what you have said above in terms that students will
> understand.  I cannot, for example, explain why your solution is more
> robust.
>

Advantages:
    Robustness; no requirement for a programmer to keep a count of
    instances or update the count of the super class by calling it's
    initializer.  This minimises the chance of programmer error.
    Avoids replication of data, eliminating OOS errors.

    Maintainability; all of the code for counting is contained in a
    single module, so modifications will not require changes to
multiple
    similar pieces of code throughout the class structure.

    Agility; we can maintain more than one zoo in a program by
creating
    new instances of of the animal_farm class.  Class level data would
    limit us to data on a single zoo system.

    Generality; the Collection class can be used for any problem of
    this type.


> >Admittedly the solution is on the complicated side, though perhaps
> >someone with more experience than me could simplify things.
> >Unfortunately, a certain amount of complexity is just a reflection of
> >the fact that your demands strain the OO paradigm right to it's limit.
> > You could possibly implement the same thing in Java with a Factory
> >pattern, and perhaps the reflection API.
>
> Your vast experience may be blinding you to the problems non-CIS
> students will have with these more complex solutions.  I may be
> pushing a paradigm to some limit, but these are real-world problems
> that should be easily solved with a good OOP language.
>

This isn't something I'd show a programming beginner either!  I'd
choose a different way of demonstrating a class heirarchy, myself.

> -- Dave
>
> >(Of course I'm none too sure I could do that after many years of
> >hacking Java vs a few weeks of Python!)
> >
> >
> >> Take a look at http://ece.arizona.edu/~edatools/Python/Exercises/ and
> >> let me know if Animals_2b.py is what you had in mind.  If not, can you
> >> edit it to show me what you mean?
> >>
> >> -- Dave