OTish: using short-term TCP connections to send to multiple slaves

[Chris Angelico]

On Sun, Nov 16, 2014 at 11:02 PM, jkn <jkn_gg at nicorp.f9.co.uk> wrote:
> I have a use case of a single 'master' machine which will need to
> periodically 'push' data to a variety of 'slave' devices on a small local
> subnet, over Ethernet. We are talking perhaps a dozen devices in all with
> comms occurring perhaps once very few seconds, to much less often - once per
> half an hour, or less. There is probably an upper bound of 64KB or so of
> data that is likely to be sent on each occasion.

This is very similar to what a MUD server has to do, with the added
advantage that you're working with ethernet (most MUDs are accessed
through the dangerous waters of the internet, and there are all sorts
of extra problems).

> Previous similar systems have attempted to do this by maintaining multiple
> long-term TCP connections from the master to all the slave devices. The
> Master is the server and the slaves periodically check the master to see
> what has changed. Although this ... works ..., we have had trouble
> maintaining the connection, for reasons ... I am not yet fully aware of.

The most likely case on the internet is that idle connections are
getting dropped, maybe after five minutes. On a LAN, that shouldn't be
happening, but can you run a test and see if maintaining the
connection (by sending dummy packets every minute, perhaps) stops the
dropouts? That might be all you need.

> We are now considering an alternative approach where the master maintains a
> list of slave devices and acts as a client. Each device is a server from the
> point of view of data transfer. We would then use a short-lived TCP
> connection when data is available; the Master would connect to each slave
> which needed the data, send it, and close the connection.

That's also possible. You may find you have a lot of overhead,
percentage-wise, but based on your figures above (a dozen devices, no
more frequent than every few seconds), I very much doubt it'll be a
problem.

> I should also add that we desire our system to be 'robust' in the face of
> situations such as cable unplugging, device power cycles, etc.

Then you want asynchronous connections. If the device at the other end
is down, don't wait for it.

> Although this might get round some of our current problems, I can see that
> we might end up with new problems to deal with. I am wondering if this
> scenario rings bells with anyone, and seeking pointers to what has been done
> elsewhere. As I say, I am expecting to prototype it in Python so any
> specifics also welcome!

Not a problem. I usually do this kind of work in Pike, rather than
Python, so I can't offer ready-to-go Python code for you, but it's
pretty similar. Is there any reason you plan just to _prototype_ this
in Python? Seems to me you could do the whole job in it, no reason to
shift to another language - unless there's a huge aspect to this
project that's computationally intensive or something. The part you've
described is going to be dominated by the network, not the programming
language.

> (suggestions as to a better forum to ask for previous experience also
> gratefully received)

I don't know of any place better. Let's keep it here, at least for the
moment (unless you switch to using Pike, in which case I'll help you
off list...). Here's how I'd do up something like you're describing,
assuming the active-server model:

class Client(object):
    def __init__(self, ip):
        self.ip = ip
        self.sock = None
        self.unsent = ""
    def send_message(self, message):
        self.unsent += message
        self.send_queued()
    def send_queued(self):
        if not self.sock:
            # attempt asynchronous establishment of socket
        else:
            # attempt to send text to the socket, asynchronously

If you're using Python 3.4, you could make use of asyncio to do most
of the work. I've not actually put it to use yet, but it looks
excellent. But the above structure is most of what I'd be looking at.
Establish connections only when you need them, and only if you don't
have them already. (Note that this assumes the other end can separate
messages out, so it's happy to combine them. You may need to make sure
you have some kind of boundary - with text, end-of-line makes good
sense.) You would then create a bunch of Clients, one for each IP you
need to connect to, and send_message() to all appropriate ones. It
should be possible to separate your async comms code from your message
generation code, and keep both of them clean.

In short: What you're doing is certainly possible, and makes some
degree of sense. (Though maintaining connections would be both easier
and simpler, if you can resolve your issues.)

ChrisA