Modern recommended exception handling practices?

[Steven D'Aprano]

Hi Vasudev, and welcome.

Sorry for the delay in replying to your questions. My answers inline, below.


On Fri, 30 Oct 2015 04:43 am, vasudevram wrote:

> Are there any modern (i.e. for current versions of Python 2 and 3)
> recommended exception handling practices?

When you say "modern", it sounds like you want to contrast them with "old
fashioned and obsolete" exception-handling practices. I don't know if there
are any obsolete exception-handling practices in Python.

I would follow these rules, as best I am able to:


(1) Remember that, in general, exception catching is very course-grained.
You can catch a particular kind of exception, but you don't know which
specific line of code or expression caused it, the reason why, or where in
the `try` block it was raised. The entire `try` clause is as fine-grained
as you get.

One consequence is that it is easy to write exceptional handling code which
is wrong and buggy:

http://blogs.msdn.com/b/oldnewthing/archive/2004/04/22/118161.aspx

http://blogs.msdn.com/b/oldnewthing/archive/2005/01/14/352949.aspx


To avoid this, the `try` block should cover the least amount of code
possible. For example, suppose you have this:

    try:
        result = calculate(mylist.index(x), func(key))
    except ValueError:
        # thrown by mylist.index if x is not found
        result = "x not found"


That code is probably unsafe, because you can't be sure that either
func(key) or calculate(...) itself might not throw a ValueError. Unless you
can guarantee that there are no circumstances where that might happen, or
if it doesn't matter which of the three expressions throws the ValueError,
you are better off writing this:

    try:
        idx = mylist.index(x)
    except ValueError:
        result = "x not found"
    else:
        result = calculate(idx, func(key))


(2) Catch only the exceptions that you expect and can genuinely recover
from. Don't use `try...except` as a way to cover up and hide the fact that
an error occurred.

One of my favourite quotes about programming is this one:

   "I find it amusing when novice programmers believe their main
    job is preventing programs from crashing. ... More experienced
    programmers realize that correct code is great, code that
    crashes could use improvement, but incorrect code that doesn’t
    crash is a horrible nightmare."
    -- Chris Smith

In context of Python, "crash" means "raise an exception". Exceptions are
your best friend: they show you were your bugs are. How can you fix bugs if
you don't know where they are?

You should only catch the exceptions that you expect to occur, and can
recover from. Everything else should be allowed to raise an exception, so
you can *see* that something is broken. Now you know that something in your
code has a bug, and needs to be fixed.

Consequently, you should never (well, *almost* never) be tempted to write a
bare `except` clause, or even `except Exception`. They are almost always
too broad, and catch too much.

Bare `except` clauses are very possibly *literally the worst* thing that you
can write in Python:

https://realpython.com/blog/python/the-most-diabolical-python-antipattern/


(3) One exception to rule (2) is that sometimes it is useful to surround
your entire main program with a `try...except`, for the purposes of logging
errors and presenting a nice clean error message to your users:

    try:
        main()
    except Exception as err:
        # Catch any and all unexpected exceptions.
        log(err)
        alert("A serious problem has occurred and MyApplication must quit. " 
              "Please contact the help desk on 999-999-999.")
        sys.exit(1)


(4) Remember that Python matches exceptions from top down, so more specific
exceptions (subclasses of a less specific exception) need to come first. So
if we write this:

    try:
        ...
    except Exception:
        print("something unexpected occurred")
    except TypeError:
        print("this is expected")


the TypeError will be caught by the first except clause, and treated as
something unexpected. Swap the order of the except clauses, and you will be
fine:

    try:
        ...
    except TypeError:
        print("this is expected")
    except Exception:
        print("something unexpected occurred")


(5) Remember that often you can avoid exceptions instead of catching
them. "Look Before You Leap" (LBYL) may be a perfectly good alternative:

    if item in mylist:
        idx = mylist.index(item)
        process(idx)
    else:
        result = "not found"


but be sensitive to the amount of work done. The above code searches the
list twice instead of just once.


(6) In general, you can consider setting up a "try" block to be practically
free in Python (which is the opposite to Java, where exception handling is
always expensive). If no exception occurs, there is virtually no difference
in speed between:

    func(x)

and

    try:
        func(x)
    except:
        pass


but if an exception does occur, actually catching it is quite costly. This
idiom is usually called "Easier to Ask for Forgiveness than Permission"
(EAFP).


(7) So which is faster, LBYL or catching the exception? That is extremely
sensitive to not just the specific operations being performed, but how
often the exceptional cases occur. In general, you must measure your code
to know.

But as a very rough rule of thumb, consider looking up a key in a dict:

    if key in mydict:
        result = mydict[key]

versus

    try:
        result = mydict[key]
    except KeyError:
        pass


In my experience, catching the KeyError is about ten times more costly than
testing for the key's presence. So if your keys are missing more than one
time in ten, it's probably better to use LBYL.


(8) Remember that this is called *exception handling*, not "error handling".
In Python, exceptions are not just for errors.

For example, Python extensively uses StopIteration for flow-control. When an
iterator runs out of items, it raises an exception, which the Python
interpreter catches to recognise the end of the iterator. You can do the
same in your own code, invent your own protocols that use exceptions to
signal exceptional cases.

Some very highly respected people like Joel Spolsky and Raymond Chen dislike
exceptions as flow-control, calling them just a thinly disguised GOTO:

http://www.joelonsoftware.com/items/2003/10/13.html

http://blogs.msdn.com/b/oldnewthing/archive/2005/01/06/347666.aspx

and while it is true that exceptions are a kind of GOTO, that doesn't mean
much. For-loops, and if-elif-else are a kind of GOTO too, and we don't
worry about them. (For example, you can't use an exception to jump *into* a
function, only out of one.)


See also:

http://c2.com/cgi/wiki?AvoidExceptionsWheneverPossible

http://c2.com/cgi/wiki?ExceptionsAreOurFriends




[...]
> I realize that there may not be any recommended standard practices for
> this, or they may be more than one set, because of differences of opinion.
> For example, several years back, there was a somewhat controversial thread
> about, IIRC, checked vs. unchecked Java exceptions, maybe triggered by
> some post on Elliot Rusty Harold's site (he used to write a lot about
> Java, both blog posts and books).

Java is, as far as I know, the only language with checked exceptions.
Certainly Python doesn't have them. I know that checked exceptions are very
controversial in the Java community, but I understand that Java's creator
James Gosling eventually came out to say that they were a mistake.

In Python's case, you should document what exceptions you expect your
functions to raise, but you must not assume that such documentation is ever
complete. For example, consider the operator "+". We know that + will raise
a TypeError:

py> 2 + "2"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

but if one of the operands is a custom type, it could raise *anything*:


py> class X:
...     def __add__(self, other):
...             raise OSError
...
py> X() + 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __add__
OSError

Does this mean you should *catch* everything? No. See Rule (2) above. Unless
you are expecting an X instance in your code, the presence of one is
probably a bug. The *unexpected* OSError will be the exception that reveals
this bug, and allows you to fix it.




-- 
Steven