OO design and multiple local sockets

[Jp Calderone]

On Fri, Jan 31, 2003 at 02:28:29AM +1300, Andrew McGregor wrote:
> 
> 
> --On Friday, January 31, 2003 00:23:35 +1100 Andrew Bennetts 
> <andrew-pythonlist at puzzling.org> wrote:
> 
> >On Fri, Jan 31, 2003 at 01:36:12AM +1300, Andrew McGregor wrote:
> >>I've got an awful case of writer's block on this one :-(
> >>
> >>Problem:  I'm writing an app (an implementation of HIP, see
> >>www.hip4inter.net) that uses raw sockets in a big way.  It needs to bind
> >>local sockets to each IP address on the system so it can transmit
> >>packets  with each source address.  It also needs to know which address
> >>packets were  received on, and from where.  So far, so good.  It's
> >>threaded (pretty much  necessarily).  There are two pools of sockets for
> >>two different protocols  that have to interact.
> >>
> >>Just the code for the protocol transforms and state machines is about 7k
> >>lines, including tests, and works, but with only a single unbound local
> >>socket, so some features I can't do yet.
> >>
> >>What I can't figure out is what a clean OO design for the socket
> >>handling  looks like, managing the various queues and worker threads
> >>without getting  really hairy.  Can anyone help?
> >
> >For OO design inspiration for socket programming, I suggest looking at
> >Twisted <http://twistedmatrix.com/> -- although admittedly it is rather
> >different to your design (being single-threaded asynchronous rather than
> >multi-threaded).  In particular its Factory and Protocol classes may be of
> >interest.
> >
> >In fact, I'd really tempted to use Twisted for this -- have you looked at
> >it?  I've found it excels for writing servers that involve multiple
> >protocols.
> >
> >(Not that being a developer of Twisted makes me biased or anything...)
> 
> It looks interesting, and I'd be perfectly happy to use the event model, 
> especially since the HIP specs are more-or-less written that way.
> 
> However, I can't see how to convince Twisted to use a raw socket.  

  Raw sockets are a transport type that aren't *directly* supported by
Twisted, but Twisted does nothing to prevent you from using them.  If you
get a CVS checkout, you'll have a sandbox/ directory.  Inside it, ipv6.py is
an example of adding a new transport (and all the associated features).  You
should be able to use this as a template for raw socket support.

> The second issue, less serious, is that some operations in the protocol
> can take several seconds of CPU time, which should ideally be done in the
> background wrt the fast actions of other connections; these can usually be
> done in a few milliseconds.  Mind you, threads don't solve that entirely
> either, because the GIL gets in the way.

  If it makes sense, you might be able to break these long operations up
into discrete parts that are each short enough to not hamper the network
operations.  Twisted provides an abstraction for this sort of thing,
Deferreds, to make it a bit easier.

  If they can't be easily broken up, threading might still be the solution. 
Twisted makes it easy to toss one function off into a thread and schedule
its return value to be delivered back to some function in the main thread.

  Jp

-- 
There are 10 kinds of people: those who understand binary and those who do
not.
-- 
 up 45 days, 21:49, 5 users, load average: 0.00, 0.00, 0.00
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20030130/ece92310/attachment.sig>