common mistakes in this simple program

[Martin A. Brown]

Greetings Ganesh,

>> You're falling into the trap of assuming that the only exception you
>> can ever get is the one that you're planning for, and then handling.
>
>Ok sure !

This point is very important, so I'll reiterate it.  I hope the poor 
horse lives.

>> ALL exceptions as though they were that one. Instead catch ONLY the
>> exception that you're expecting to see, and ignore everything else. Do
>> not use a bare "except:" clause, nor even "except Exception:", for
>> this. You will appreciate it later on.
>
>What option do I have now in so ensure that I loop over the For 
>loop. 

>Try except is ruled out ?

No, no not at all!

  #1:  Yes, you (probably) should be using the try-except construct.
  #2:  No, you should not (ever) use a bare try-except construct.

Please read below.  I will take a stab at explaining the gaps of 
understanding you seem to have (others have tried already, but I'll 
try, as well).

I am going to give you four different functions which demonstrate 
how to use exceptions.  You may find it instructive to paste these 
functions into an interactive Python shell and try them out, as 
well.

I will explain a few different cases, which I will show below, but I 
think you'll need to figure out how to apply this to your situation.


Step 1: catch a specific Exception
----------------------------------

  def catchValueError(val, default=0):
      '''somebody passed in a non-convertible type, return a default'''
      try:
          return int(val)
      except ValueError:
          return default

Now, let's try passing a few values into that function:

  >>> catchValueError(0)
  0
  >>> catchValueError('-3')
  -3
  >>> catchValueError('Catullus')
  0

The last example above is the most interesting.  We fed a string to 
the builtin function int() and it raised a ValueError.  Our little 
try-except block, however, caught the ValueError and performed our 
desired action (in this case, returning a default value).


Step 2: catch a specific Exception
----------------------------------

But, what about this:

  >>> catchValueError(dict())
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 4, in catchValueError
  TypeError: int() argument must be a string or a number, not 'dict'

Now, you can see that there's a different problem.  The exception is 
different if we feed different data to the function.  So, let's try 
again, but maybe we catch the TypeError instead.

  def catchTypeError(val, default=0):
    '''somebody passed in a non-convertible type, return a default'''
    try:
        return int(val)
    except TypeError:
        return default

This seems to work--but, now, if we try our other values (which 
worked before), they don't work now:

  >>> catchTypeError(dict())
  0
  >>> catchTypeError(0)
  0
  >>> catchTypeError('-3')
  -3
  >>> catchTypeError('Catullus')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 4, in catchTypeError
  ValueError: invalid literal for int() with base 10: 'Catullus'


Step 3: catch several different classes of Exception
----------------------------------------------------

You can a single try-except block to catch multiple classes of 
specific Exception (and then perform exception handling).

  def catchTVError(val, default=0):
    '''somebody passed in a non-convertible type, return a default'''
    try:
        return int(val)
    except (ValueError, TypeError):
        return default

In this little toy demonstration, you can see that this is probably 
the desired functionality.

  >>> catchTVError(dict())
  0
  >>> catchTVError(0)
  0
  >>> catchTVError(-3)
  -3
  >>> catchTVError('Catullus')
  0


Step 4: catch many different classes of Exception
-------------------------------------------------

Now, suppose you want to deal with each possible exception in a different
manner....you can have different exception-handling behaviour for each class of
exception.

  def altCatchTVError(val, default=42):
      '''somebody passed in a non-convertible type, return a default'''
      try:
          return int(val)
      except ValueError:
          return abs(default)
      except TypeError:
          return 0 - abs(default)

As you can see, this offers a good deal more flexibility for handling specific
exceptions that you might encounter.

  >>> altCatchTVError(0)
  0
  >>> altCatchTVError(22)
  22
  >>> altCatchTVError('17')
  17
  >>> altCatchTVError('-3')
  -3
  >>> altCatchTVError('str')
  42
  >>> altCatchTVError(dict())
  -42


Interlude and recommendation
----------------------------

As you can see, there are many possible Exceptions that can be raised when you
are calling a simple builtin function, int().

Consider now what may happen when you call out to a different program; you
indicated that your run() function calls out to subprocess.Popen().  There are
many more possible errors that can occur, just a few that can come to my mind:

  * locating the program on disk
  * setting up the file descriptors for the child process
  * fork()ing and exec()ing the program
  * memory issues
  * filesystem disappears (network goes away or block device failure)

Each one of these possible errors may translate to a different exception.  You
have been tempted to do:

  try:
      run()
  except:
      pass

This means that, no matter what happens, you are going to try to keep
continuing, even in the face of massive failure.

To (#1) improve the safety of your program and the environments in 
which it operates, to (#2) improve your defensive programming 
posture and to (#3) avoid frustrating your own debugging at some 
point in the future, you would be well-advised to identify which 
specific exceptions you want to ignore.

As you first try to improve the resilience of your program, you may 
not be certain which exceptions you want to catch and which 
represent a roadblock for your progam.  This is something 
that usually comes with experience. 

To get that experience you can define your own exception (it'll 
never get raised unless you raise it, so do not worry).  Then, 
create your try-except block to catch only that one.  As you 
encounter other exception that you are certain you wish to handle, 
you can do something with them:

  class UnknownException(Exception):
      pass


  def prep_host():
      """
      Prepare clustering
      """
      for cmd in ["ls -al",
                  "touch /tmp/file1",
                  "mkdir /tmp/dir1"]:
          try:
              if not run_cmd_and_verify(cmd, timeout=3600):
                  logging.info("Preparing cluster failed ...")
                  return False
          except (UnknownException,):
              pass
      logging.info("Preparing Cluster.....Done !!!")
      return True

Now, as you develop your program and encounter new exceptions, you 
can add new except clauses to the above block with appropriate 
handling, or (re-)raising the caught exception.


Comments on shelling out to other programs and using exceptions 
--------------------------------------------------------------- 
Exceptions are great for catching logic errors, type errors, 
filesystem errors and all manner of other errors within Python 
programs and runtime environments.  You introduce a significant 
complexity the moment you fork a child (calling subprocess.Popen).

It is good, though, that you are testing the return code of the 
cmd that you pass to the run() function.


Final advice:
-------------
Do not use a bare try-except.

You will frustrate your own debugging and your software may end up 
trying to excecute code paths (or external programs, as you are 
doing right now) for which you were sanity checking.

-Martin

-- 
Martin A. Brown
http://linux-ip.net/