GeneratorExit should derive from BaseException, not Exception

[Chad Austin]

Hi all,

First, I'd like to describe a system that we've built here at IMVU in 
order to manage the complexity of our network- and UI-heavy application:

Our application is a standard Windows desktop application, with the main 
thread pumping Windows messages as fast as they become available.  On 
top of that, we've added the ability to queue arbitrary Python actions 
in the message pump so that they get executed on the main thread when 
its ready.  You can think of our EventPump as being similar to Twisted's 
reactor.

On top of the EventPump, we have a TaskScheduler which runs "tasks" in 
parallel.  Tasks are generators that behave like coroutines, and it's 
probably easiest to explain how they work with an example (made up on 
the spot, so there may be minor typos):

	def openContentsWindow(contents):
		# Imagine a notepad-like window with the URL's contents...
		# ...

	@threadtask
	def readURL(url):
		return urllib2.urlopen(url).read()

	@task
	def displayURL(url):
		with LoadingDialog():
			# blocks this task from running while contents are being downloaded, 
but does not block
			# main thread because readURL runs in the threadpool.
			contents = yield readURL(url)

		openContentsWindow(contents)

A bit of explanation:

The @task decorator turns a generator-returning function into a 
coroutine that is run by the scheduler.  It can call other tasks via 
"yield" and block on network requests, etc.

All blocking network calls such as urllib2's urlopen and friends and 
xmlrpclib ServerProxy calls go behind the @threadtask decorator.  This 
means those functions will run in the thread pool and allow other ready 
tasks to execute in the meantime.

There are several benefits to this approach:

1) The logic is very readable.  The code doesn't have to go through any 
hoops to be performant or correct.
2) It's also very testable.  All of the threading-specific logic goes 
into the scheduler itself, which means our unit tests don't need to deal 
with any (many?) thread safety issues or races.
3) Exceptions bubble correctly through tasks, and the stack traces are 
what you would expect.
4) Tasks always run on the main thread, which is beneficial when you're 
dealing with external objects with thread-affinity, such as Direct3D and 
Windows.
5) Unlike threads, tasks can be cancelled.

ANYWAY, all advocacy aside, here is one problem we've run into:

Imagine a bit of code like this:

	@task
	def pollForChatInvites(chatGateway, userId, decisionCallback, 
startChatCallback, timeProvider, minimumPollInterval = 5):
		while True:
			now = timeProvider()
	
			try:
				result = yield chatGateway.checkForInvite({'userId': userId})
				logger.info('checkForInvite2 returned %s', result)
			except Exception:
				logger.exception('checkForInvite2 failed')
				result = None
			# ...
			yield Sleep(10)

This is real code that I wrote in the last week.  The key portion is the 
try: except:  Basically, there are many reasons the checkForInvite2 call 
can fail.  Maybe a socket.error (connection timeout), maybe some kind of 
httplib error, maybe an xmlrpclib.ProtocolError...  I actually don't 
care how it fails.  If it fails at all, then sleep for a while and try 
again.  All fine and good.

The problem is that, if the task is cancelled while it's waiting on 
checkForInvite2, GeneratorExit gets caught and handled rather than 
(correctly) bubbling out of the task.  GeneratorExit is similar in 
practice to SystemExit here, so it would make sense for it to be a 
BaseException as well.

So, my proposal is that GeneratorExit derive from BaseException instead 
of Exception.

p.s. Should I have sent this mail to python-dev directly?  Does what I'm 
saying make sense?  Does this kind of thing need a PEP?

-- 
Chad Austin
http://imvu.com/technology