xmlprclib/server not reusing connections

[Andrew Bennetts]

On Mon, Feb 23, 2004 at 09:57:55PM -0800, Roger Binns wrote:
> > You might have better luck with Twisted http://twistedmatrix.com/.
> 
> I did look at twisted but really don't like it.  In particular I really
> don't like the event driven code, that has to add callbacks and
> deal with partial state when called.

I was initially skeptical too, but I found that the small cost of using
callbacks to be well worth the benefit of avoid threads most of the time.
Threads invite all sorts of difficult to reproduce and debug problems like
race conditions and deadlocks.

That said, event-driven programming doesn't suit every problem.  Twisted
does try fairly hard to reduce the burden of asynchronous code, though:
Deferreds make chains of callbacks easy to handle, and the twisted.flow
package uses python 2.2's generators to allow synchronous-looking code to
yield to the event loop:
    http://twistedmatrix.com/documents/howto/flow

And of course Twisted's splitting of network code into reactor, factory,
protocol and transport objects is a really nice abstraction, but that would
probably translate just fine into blocking code too (except perhaps the
reactor would be largely irrelevant?).

> That is certainly the right way to do things if you want to avoid
> using threads.  It is however complicated and convoluted.  See this
> example:
> 
> http://twistedmatrix.com/documents/howto/tutorial#auto20

That example looks fine to me, but perhaps I've been using Twisted too long!
:)

I certainly have no problems reading that code and immediately understanding
what every part of it does.

> The model I far prefer is to use multiple threads (available on all
> major Python platforms) and use worker threads with work queues.
> They turn out to be simpler since they don't have queue callbacks
> or effectively be a glorified state machine scattered across
> several functions.

That's fine.  Twisted doesn't prevent you from using threads!
reactor.callInThread(func, args...) will run a function inside a thread pool
(or you can just start and manage threads yourself), and
reactor.callFromThread provides a safe way for a thread to run some code in
the main event loop.  Twisted itself does this in e.g. the
twisted.enterprise.adbapi module, to provide an asynchronous interface
around a synchronous DB-API modules.

> > Glancing at that, it looks like using Twisted to implement that would be a
> > lot easier, and shorter.
> 
> Except it wouldn't unless twisted already had the necessary functionality
> which it doesn't.  I would have to go through a similar exercise with
> twisted, which is far more complicated.

Well, Twisted already has persistent HTTP connections and SSL.  So e.g. the
vast bulk of your XMLRPCRequestHandler.do_POST would disappear, because
twisted.protocols.http already does that.

Perhaps my point wasn't clear: I wasn't saying that Twisted will inherently
make any network code much much shorter[1], I was saying that Twisted
already has sufficient HTTP and SSL support to meet your needs, and that you
could re-use that.  Of course, now that you've written what you need without
Twisted, you probably don't care very much :)

-Andrew.

[1] Although I think on average it probably does, and makes it easier to
    write, too.  The wide variety of protocols available in twisted.protocols
    suggests to me that I'm at least part right about this...