Python threading?

[Andrew Bennetts]

Robert Oschler wrote:
> Having written _many_ state machines in my life, which async programming
> boils down to in one form or another, I feel it depends on the task.  When
> you have an async event driven model, the code that services a
> "transaction-package" ends up spreading functionality that services the
> "transaction-package" application's code landscape.  This can frequently
> be a headache.  As an example of a "transaction-package", take all the
> steps involved in servicing an incoming TCP connection, with custom
> authentication of a user, and perhaps a header and data object
> transacttion or two, finishing with a proper ending and cleanup of the
> transaction, and with subsequent notification of all necessary modules and
> data structures of the results of the transaction.  For dessert add a
> centralized clean error handler that can log errors propertly and
> gracefully recover the app regardless of where in the distributed event
> flow the error took place.

You want Twisted.  http://twistedmatrix.com/

It's an asynchronous networking framework which has an elegant abstraction
of callbacks and error handlers that I find make event-based programming a
breeze.  See http://twistedmatrix.com/documents/howto/defer for a detailed
description of how this works.  It makes it trivial to sequence a series of
operations on some event, regardless of when it happens.

In fact, here's a breakdown of your steps and Twisted:

1. "servicing an incoming connection": 
  twisted.internet does that for you.  It's dead easy.

2. "custom authentication of a user":
  Use the twisted.cred package.  This is exactly what it is designed for.
  It has a standard asynchronous interface, with ready-made implementations
  involving a simple python dict, and also a database-based implementation.

3. "perhaps a header and a data object transacttion":
  There isn't general transaction support in Twisted, but I imagine any
  object database you'd be using would support them, e.g. the ZODB.  And
  again, Deferreds make it easy to catch exceptions and so on (see my next
  points).

4. "centralized clean error handler than can log errors properly and
   gracefully recover":
  Twisted's Deferreds make that trivial.  In fact, if you forget to install
  an error handler, Twisted will log the error for you automatically.
  Deferreds callbacks and errbacks in many ways mirror your usual
  try/except constructs, so structured error handling is straightforward.
  See the doc I linked to above for more details.  Also, Twisted's main loop
  will never be crashed by an exception in one of the connections it
  services... in general, if something causes one connection to crash, the
  rest of Twisted merrily keeps on going regardless.

> Contrast this with a simple procedure call where a linear series of
> function calls are made, each with appropriate error handling and status
> adjustment, in a pleasant centrally located manner.

For instance, talking to a database asynchronously, Twisted-style:

    # See also http://twistedmatrix.com/documents/howto/enterprise
    deferred = db.runQuery('SELECT * FROM table')

Then hooking up a bunch of callbacks:
    deferred.addCallbacks(gotResult, somethingBroke)
    deferred.addCallbacks(doStep2, step2Rollback)
    deferred.addCallback(doStep3)   # step 3 doesn't care about rollbacks
    # ...etc...
    
    # Catch any uncaught errors
    deferred.addErrback(logError)

The results of each callback get fed into the next, so you can perform a
series of data transformations or whatever, e.g. perhaps you want something
like:
    # Note that addCallback returns the Deferred for your convenience
    deferred.addCallback(recordSetToXML).addCallback(applyXSLT)  # etc

Which looks like a linear series of function calls to me, and it certainly
has error handling (.addErrback) and status adjustment (you could easily be
updating an instance variable from within the callbacks).

And there's even more to Deferreds than I've described here!  Go read the
documentation :)

> There is an upcoming massively multiplayer game called EVE which is using
> Stackless Python which makes this point quite well.  Game AI suffers from
> the same design considerations as mentioned above.  Here's the link if
> you're interested (I'm not associated with them in any way, just
> considering taking a look at Stackless).  Look for question "8.5 How is
> the game logic implemented in EVE?":

Interesting.  Twisted is actually designed with exactly this goal in mind
(MMORPG).  Stackless is nice, but I think for what you're talking about
Twisted already solves your problems in "pure" Python :)

Twisted is an excellent framework.  I highly recommend you download it
(version 0.99.2 is still fresh off the press -- grab it while it's hot!),
play with it, bug us on the mailing list and irc channel (#twisted on
irc.freenode.net), and let us know what you think.  We think you might just
like it.

-Andrew.