PythonD: 4Suite or Twisted?

[David Bolen]

Moshe Zadka <m at moshez.org> writes:

> On 04 Jun 2003, David Bolen <db3l at fitlinxx.com> wrote:
> 
> > Just one small comment, since I generally don't see it mentioned in
> > postings about Twisted - the underlying protocol support for internet
> > components appear robust in terms of TCP support, but we found that
> > UDP support was much weaker when we last looked at Twisted (not too
> > long ago).
> 
> These kinds of comments really should exact numbers. UDP was specifically
> marked "unstable" in Twisted 1.0. Since then it has been revamped, and
> seems to be working pretty well. Do you have any specific lacks or
> complaints about UDP in Twisted?

I'm not sure that making a comparison between robust and weaker
necessarily requires numbers (unless you're saying that UDP support in
Twisted really isn't weaker than TCP), since it's just intended to be
a general comment and not a set of metrics to base a decision on.

But since you asked :-)

First, I'll re-iterate that I'm not knocking Twisted - it's an
impressive framework.

However, the last time we specifically tried Twisted out for one of
our applications - a distributed processing system - was probably
around January/February - it's true that this was probably 1.0 (or
0.99.4 which was really close to 1.0).  At that time, a big item that
showed up was that the perspective broker, a key reason for looking at
Twisted, didn't support UDP.  The developer doing the testing also had
problems getting some of the higher level objects and/or samples to
function when trying to convert them from TCP to UDP, including some
issues with the IReactorUDP class (sorry, but I don't have hard notes
on that handy).  We then noticed the various UDP classes generally
marked as unstable, and figured they weren't quite as well supported
within the core framework.  I assumed it was most likely due to lack
of real need by current Twisted users, which I'm certainly not
complaining about, but it wasn't worth our pursuing, since we had our
own code base we could also work from as an alternative.

I checked back last month (version 1.0.4) and saw the
twisted.internet.udp class was still marked as "unstable" in the
documentation and IReactorUDP as semi-stable.  This still seems true
with 1.0.5 on the web site, and there aren't many UDP related comments
in the change log.

While reviewing the documentation (then and now) I think one item that
struck me is the protocol structure where just about everything lies
beneath the twisted.internet.protocol.Protocol class (which inherits
from twisted.internet.protocol.BaseProtocol), and that class is often
used as central to higher level functionality.  And the peer class
twisted.internet.protocol.ProcessProtocol also inherits from
BaseProtocol.

Stuff layered above this really seems geared towards this Protocol
class - for example, twisted.internet.protocol.Factory is designed to
create subclasses of Protocol, and the Factory is what is used by many
higher level system objects like the perspective broker.  True, I
don't think the implementation of Factory prevents you from using a
class not beneath Protocol, but then the users of the factory will
generally blow up when they try to use a method in the Protocol
interface that might not be present.

All of this is really nice and well-structured, at least without UDP.
UDP is twisted.internet.protocol.DatagramProtocol, but that doesn't
even inherit from BaseProtocol, much less be a subclass of protocol.
And because it's not a subclass of Protocol it doesn't get to play
equally in a lot of the higher level support of Twisted.  Once you
start getting into the higher order framework objects that layer on
top of all of the factory and protocol work beneath, UDP starts to
feel like the odd man out.

As a trivial example, take the "echoserv.py" example.  I can't see why
that example can't become a UDP example just by changing the
"listenTCP" call to "listenUDP".  However, because the example server
is a subclass of Protocol, and the UDP DatagramProtocol class is going
to call "datagramReceived" rather than Protocol's "dataReceived" it
doesn't work.  But there's no reason it couldn't, since at that level
of abstraction, UDP and TCP could be treated equally.  Now in recent
versions of Twisted, there is a new echoserv_udp.py example whose sole
difference is to inherit from DatagramProtocol and change the single
method in the server to datagramReceived.

I think my conclusions at the time were that the Protocol class was
really a StreamProtocol class in disguise, and there was probably no
top level "protocol" abstraction in the system.  Datagram and Stream
protocols can be like night and day at the protocol level, but there's
also a lot of commonality that ought to be able to permit some of the
higher level framework functionality to work in either case.

The documentation doesn't tend to highlight TCP versus UDP
differences, but just uses the factory and protocol level, so it's not
generally clear where something won't work in both cases.  I just
ended up feeling like trying to use UDP for a complex application with
Twisted (as opposed to a dedicated UDP protocol implementation like
DNS) was swimming against the tide.  I think that's still a valid
state of affairs at the moment.

> (Note that currently Twisted.Names is an authoritative DNS for a domain
> with no problems, and DNS is a UDP-based protocol)

Right, it's a UDP based _protocol_, but not a UDP based "application",
so it doesn't need much from Twisted other than it's pure asynchronous
message loop.  For me, that wasn't the attraction and potential of
Twisted for us (we have plenty of async loops of our own around :-)),
but the support layers above that all working together.

-- David