exception handling in complex Python programs

[Steven D'Aprano]

On Wed, 20 Aug 2008 17:49:14 -0700, dbpokorny at gmail.com wrote:

> On Aug 20, 10:59 am, Steven D'Aprano <st... at REMOVE-THIS-
> cybersource.com.au> wrote:
>> Oh goodie. Another programmer who goes out of his way to make it hard
>> for other programmers, by destroying duck-typing.
> 
> Remember kids: personal attacks are cruise control for cool.

It might not be enjoyable to have a sarcastic remark directed your way, 
but it isn't a personal attack. Just because a comment is about something 
you do doesn't make it a personal attack. Personal attacks are about who 
you are rather than what you do.


> So this was a simplification - most of the asserts I've written don't
> actually use isinstance, partly because typing isinstance takes too
> long. 

You say that you liberally sprinkle isinstance() checks through your 
code, then you say that you don't. That confuses me.

There's an apparent contradiction in your argument. You seem to be 
arguing against EAFP and in favour of LBYL, but now you're suggesting 
that you don't use type-checking. As near as I can tell, you don't do 
type-checking, you don't do duck typing, you don't like catching 
exceptions. So what do you actually do to deal with invalid data?

By the way, if you're worried that isinstance() is too long to type, you 
can do this:

isin = isinstance
isin(123, int)



> The point is to create a barricade so that when something goes
> wrong, you get an assertion error against the code you wrote
...
> assert statelt.tag == 'stat'
> assert len(path) > 0 and path[0] == '/' 
> assert self.__expr != None

All of those examples seem to be reasonably straight forward tests of 
program logic, which would make them good cases for assertions. Assuming 
that statelt, path and self.__expr are internally generated and not user-
supplied arguments.

I'll note that testing for (non-)equality against None is possibly a 
mistake. It won't catch the case where self.__expr is an object that, for 
some reason, compares equal to None but isn't None. If that's your 
intention, then it's fine, but given that you don't seem to be a big fan 
of duck typing I guess you'd probably be better off with:

assert self.__expr is not None

It runs faster too, although a micro-optimization of that tiny size isn't 
in itself sufficient reason for preferring "is not" over "!=". The real 
reason is to avoid accidental matches.


> So here asserts are used to made distinctions that are more fine-
> grained than type.

The problem with your earlier example isn't that isinstance() is too 
coarse-grained (although I try to avoid it as much as possible), but that 
assert is not meant for argument testing. In principle, the end user 
should never see an AssertionError. As I said earlier, assert is for 
testing program logic.


>> > and other similar assertions in routines. The point is that EAFP
>> > conflicts with the interest of reporting errors as soon as possible
>>
>> Not necessarily. Tell me how this conflicts with reporting errors as
>> soon as possible:
>>
>> def do_something(filename):
>>     try:
>>         f = open(filename)
>>     except IOError, e:
>>         report_exception(e)  # use a GUI, log to a file, whatever...
>>
>> How could you report the exception any earlier than immediately?
> 
> Here is an example: 

I gather by your lack of answer to my question that you now accept that 
exceptions don't necessarily delay reporting errors as early as possible.


> a simple query tool for a tiny "mock SQL" relational
> database. With a method (called "select_all") you can perform the
> equivalent of a select query on the database. The contents of the query
> are specified with triples of the form
> 
> [field, comparison_operator, value]
> 
> for instance ['name', operator.equals, cmd_name]. You can also specify
> an "order by" field which is None by default. In the code written, there
> is an assertion that the order-by field is either None or a valid field
> name (we can't order by a nonexistent field!). If the assertion isn't
> there, then I will get an error on this line:
> 
>   key_extractor = KeyExtractor(q_column_names.index(order_by_column))
> 
> In this particular case, I will get a ValueError (what does ValueError
> mean again?

It means you've supplied an invalid value. What did you think it meant?

In fact, what you get is:

ValueError: list.index(x): x not in list

which tells you exactly what went wrong and why. It's concise and self-
explanatory.


> And what is this KeyExtractor?) 

Irrelevant. That's not part of the exception. It just happens to be on 
the same line of source code as the expression that raises an exception.


> since the index method will
> fail. I wrote the tiny relational database a long time ago, and I really
> don't want to put pressure on my mental cache by thinking about the
> internal logic of this chunk of code. After scratching my head for a
> while, I'll probably figure it out. Now imagine that you instead get an
> error on this line:
> 
>   assert order_by_column in q_column_names
> 
> Now the programming error slaps me with a fish and yells "STOP! YOU
> CAN'T ORDER BY A FIELD THAT DOESN'T EXIST!!!". It will take about 2
> seconds to figure out what went wrong. 

You're assuming that, six months from now, you'll see this error:

>>> q_column_names = ['fee', 'fi', 'fo', 'fum']
>>> order_by_column = 'floop'
>>> assert order_by_column in q_column_names
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

and remember that the reason you were testing the assertion is that 
*later on* you intend to order a field by something named 
"order_by_column". Or maybe order_by_column is what you've *already* 
ordered the field by, and now you're about to do something else? Who 
knows what it means? It could mean anything.

To make your code actually useful, you need to supply an assertion string 
that explains *why you care* about the assertion, and even then you 
better not rely on it because asserts can be turned off at runtime and 
the test might not even be executed.



> I just saved a minute figuring
> out what the problem is. Multiply that by ten, and you've just
> eliminated work in a potentially laborious debugging session.

I'm not convinced by your example. Your example seems to actually make 
debugging harder, not easier.


> If you look at the history of the EAFP concept in Python, then you see
> that it comes from Alex Martelli's Python in a Nutshell around pages
> 113-114. 

I think it actually comes from a lot earlier than Alex's book.


> I don't think the code examples make the case for EAFP very
> well (not that I know what EAFP is in the first place, given that it is
> barely explained. 

Deary me. Perhaps you should find out what it is before critiquing it?

> I interpret it as "wrap questionable stuff in try/
> except blocks"), 

As a one-line summary, that's not bad.

> and in any case there is practically no support for
> using EAFP as the dominant error-handling paradigm. 

You're joking, right? You're trying to wind me up for a laugh?

What exactly do you think try...except... blocks are for, if not EAFP?


> If you look at Code
> Complete, then you'll see the opposite suggestion, namely that
> exceptions should only be used for truly exceptional circumstances (i.e.
> bugs). 

No no no, exceptions are not necessarily bugs!!! A bug is an exceptional 
circumstance, but not all exceptional circumstances are bugs.


>>> list_of_cheeses = "Wensleydale Mozzarella Stilton Edam Feta"
>>> list_of_cheeses.index('Cheddar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: substring not found


That's not a bug, that just means that Cheddar isn't found in the list of 
cheeses. The bug is *failing to deal with the exceptional circumstances*.

>>> try:
...     list_of_cheeses.index('Cheddar')
... except ValueError:
...     print "Sorry sir, we don't sell cheddar. It's not very popular."
... else:
...     print "Yes, we have some cheddar."
...
Sorry sir, we don't sell cheddar. It's not very popular.


I think I can summarize the rest of your post as:

* EAFP isn't a panacea for everything.

* badly written code can abuse try...except blocks.

* if you run a single-tasking architecture where nothing is shared, then 
race conditions aren't a problem.

I certainly won't argue with the first two.



-- 
Steven