File handling: The easy and the hard way

[Steve Holden]

Jeremy Jones wrote:

> Hans-Joachim Widmaier wrote:
> 
[...]
> 
>> Especially not when the error is not in the program itself, but rather
>> just a mistyped filename. (Most of my helper scripts that we use to
>> develop software handle files this way. And even my co-workers don't
>> recognize 'file or directory not found' for what it is.) End users are
>> entitled to error messages they can easily understand like "I could not
>> open 'blaah' because there is no such file".
> 
> So, you're saying that dumping a raw traceback like:
> IOError: [Errno 2] No such file or directory: '/foo/bar/bam'
> to a logfile is a no-no?  Instead, it should say:
> 
> I'm sorry sir, but an error occurred when trying to write to file 
> /foo/bar/bam because it wasn't there.
> 
A traceback is not an error message. When a non-programming user sees a 
fifteen-line stack trace with python statements and line numbers in it 
that's quite enough to stop most of them from even reading it to see if 
there's anything they understand at all.

> I think the traceback is perfectly understandable.  I think that even an 
> end-user would be able to comprehend that type of message.  Or, if you 
> get an IOError, is this not sufficient:
> IOError: [Errno 28] No space left on device
> ?
> 
Again, that's not a traceback. It's an error message.

> Chances are, end users aren't going to be particulary concerned with 
> exceptions you log - unless they've got a problem that they can't figure 
> out.  And if they've got a problem they can't figure out, you'd be 
> better off giving them as much information as you can give them, or 
> they'll come to you for help.  And when they do come to you for help, 
> you'd better make sure you've given yourself the most informatino you 
> can to solve the problem.  So logging a traceback is a great idea IMHO.  
> Now, in areas where you're dead sure that an exception is nothing to be 
> concerned with, don't bother.  So, a good approach may be: handle the 
> specific exceptions that you know may occur, let other unexpected (or 
> expected in worst case scenarios) exceptions filter up to a higher 
> level, log them there, and if need be, terminate program execution.
> 
This isn't about not terminating the program, it's about reporting the 
reasons in a manner acceptable to average users.

>> Graceful error handling is
>> even more important when a program isn't just run on a command line but
>> with a GUI.
>>  
>>
> Maybe so.  But if you hit an "Oh, crap, what do I do now?" exception, 
> you may want to throw up a dialog box with a traceback or something and 
> when the user clicks OK on it, terminate program execution.  That gives 
> them a chance to (unlikely) figure out what they can do to remedy the 
> situation, otherwise call for help.
> 
I'm all for LOGGING tracebacks. Indeed WingIDE is a beauty in this 
respect, since it's also prepared to send feedback to Wing if you ask it 
to, as is Mozilla and (nowadays) Internet Explorer.

Given this, there's little excuse for showing the traceback in the 
regular case, though I don't object to allowing users to look for it if 
they want.

>> Which means? Which means that all this convenient file handling that
>> Python offers really should not be used in programs you give away. When I
>> asked for a canonical file access pattern some months ago, this was the
>> result:
>> http://groups.google.com/groups?hl=de&lr=&ie=UTF-8&threadm=pan.2003.12.30.21.32.37.195763%40web.de&rnum=1&prev=/groups%3Fhl%3Dde%26lr%3D%26ie%3DUTF-8%26q%3Dfile%2Bpattern%2Bcanonical%26btnG%3DSuche%26meta%3D 
>>
>>
>> Now I have some programs that read and write several files at once. And
>> the reading and writing is done all over the place. If I really wanted to
>> do it "right", my once clear and readily understandable code turns into a
>> nightmare. This doesn't look like the language I love for its clarity and
>> expressivness any more. Python, being a very high level language, needs a
>> higher level file type, IMHO. This is, of course, much easier said than
>> done. And renown dimwits like me aren't expected to come up with 
>> solutions.
>> I've thought about subclassing file, but to me it looks like it wouldn't
>> help much. With all this try/except framing you need to insert a call
>> level anyway (wondering if this new decorator stuff might help?). The 
>> best
>> I've come up so far is a vague idea for an error callback (if there isn't
>> one, the well known exceptions might be raised) that gets called for
>> whatever error occured, like:
>>
>> class File:
>>    ...
>>    def write(self, data):
>>        while True:
>>            try:
>>                self._write(data)
>>            except IOError, e:
>>                if self.errorcallback:
>>                    ret, dat = self.errorcallback(self, F_WRITE, e, data)
>>                    if ret == F_RETURN:
>>                        return dat
>>                else:
>>                    raise
>>
>> The callback could then write a nice error message, abort the program,
>> maybe retry the operation (that's what the 'while True'-loop is for) or
>> return whatever value to the original caller. Although the callback
>> function will usually be more than a few lines, it can be reused. It can
>> even be packed into your own file-error-handling module, something the
>> original usage pattern can't.
>>  
>>
> Hmmm....interesting.  Shouldn't you put a break after your 
> self._write(data)?  This is probably not a bad way of going about 
> things, but what types of files are we talking about here?  Log files?  
> I think you're probably better off using the builtin logging and just 
> dump raw tracebacks in there.  Data files?  Then you've probaby got that 
> wrapped in code to write formatted data to the data file anyway in which 
> case, this type of specialized class is probably not a bad thing.  If 
> you're trying to write data to a data file, you don't want litter it 
> with error messages.  You want to log it and, maybe even unlink the data 
> file and do something special.
> 
>> If you still bear with me, you might as well sacrifice a few more seconds
>> and tell me what you think about my rant. Is everything just fine as 
>> it is
>> now? Or do I have a point? I always felt it most important to handle all
>> errors a program may encounter gracefully, and the easier this is to do,
>> the less likely it is a programmer will just sneak around the issue and
>> let the interpreter/run time system/operating system handle it. (And yes,
>> I'm guilty of not obeying it myself, as it can double or triple the time
>> needed to write the whole program; just because its so cumbersome.)
>>
>>  
>>
> I dunno - something just doesn't feel right here.  I kinda feel like 
> you're wanting to create an over-generalized solution.  Your File class 
> is interesting and may be a good start for a lot of general solutions 
> and having a callback mechanism helps specialize it, but....something 
> just doesn't sit totally right here with me.  This may work totally 
> perfectly and may be an excellent piece of code to handle all of your 
> file writing activities.  I dunno....
> 
> You're not going to be able to catch every exception - not meaningfully, 
> anyway.  You could do something like:
> 
> if __name__ == "__main__":
>    try:
>        main()
>    except Exception, e:
>        log(e)
> 
> But that isn't handling all errors....
> 
It's certainly catching all subclasses of Exception, though, which in 
modern Python should be everything not handled inside (string exceptions 
are a throwback, retained for compatibility reasons). As to whether they 
are bing "handled", I guess that's a matter of opinion.

> Production quality code doesn't necessarily mean never terminating 
> because of an exception.  You want to reduce the frequency of program 
> termination due to exceptions.  I can appreciate your desire to make 
> sure you've got good solid software, and not encumber the end user with 
> ever little exception you hit, but sometimes it's OK to log/show 
> exceptions.  Like I said earlier, when you hit an exception, you hit it 
> for a reason.  Do your best to try to figure out what that reason is, 
> deal with it, figure out the most reasonable thing to do with _that_ 
> exception, and move on.  Sometimes that'll mean throwing a traceback to 
> a log file, sometimes it will mean handling it gracefully and "prettying 
> up" the message for logging or display to the end user, sometimes it 
> will mean totally ignoring it, other times you may need to just stop the 
> program.  All of these resolutions can be part of a production quality 
> piece of software.  The discerning programmer has to decide which 
> solution is appropriate for which situation.  Like Steve Holden 
> mentioned, it's really good that you're concerned with such things, but 
> make sure you apply common sense to each scenario.
> 
>> Waiting-for-you-to-jump-on-me'ly yours,
>> Hans-Joachim
>>  
>>
> Hope I didn't jump too hard.
> 
Possibly.

> Jeremy Jones

regards
  Steve