Developing a network protocol with Python

[Tom Anderson]

On Mon, 12 Dec 2005, Laszlo Zsolt Nagy wrote:

> I think to be effective, I need to use TCP_NODELAY, and manually 
> buffered transfers.

Why?

> I would like to create a general messaging object that has methods like
>
> sendinteger
> recvinteger
> sendstring
> recvstring

Okay. So you're really developing a marshalling layer, somewhere between 
the transport and application layers - fair enough, there are a lot of 
protocols that do that.

> To be more secure,

Do you really mean secure? I don't think using pickle will give you 
security. If you want security, run your protocol over an TLS/SSL 
connection.

If, however, you mean robustness, then this is a reasonable thing to do - 
it reduces the amount of code you have to write, and so reduces the number 
of bugs you'll write! One thing to watch out for, though, is the 
compatibility of the pickling at each end - i have no idea what the 
backwards- and forwards-compatibility of the pickle protocols is like, but 
you might find that if they're on different python versions, the ends 
won't understand each other. Defining your own protocol down to the 
bits-on-the-socket level would preclude that possibility.

> I think I can use this loads function to transfer more elaborate python 
> stuctures:
>
> def loads(s):
>   """Loads an object from a string.
>     @param s: The string to load the object from.
>   @return: The object loaded from the string. This function will not 
> unpickle globals and instances.
>   """
>   f = cStringIO.StringIO(s)
>   p = cPickle.Unpickler(f)
>   p.find_global = None
>   return p.load()

I don't know the pickle module, so i can't comment on the code.

> Am I on the right way to develop a new protocol?

Aside from the versioning issue i mention above, you should bear in mind 
that using pickle will make it insanely hard to implement this protocol in 
any language other than python (unless someone's implemented a python 
pickle library in it - is there such a beast for any other language?). 
Personally, i'd steer clear of doing it like this, and try to use an 
existing, language-neutral generic marshalling layer. XML and ASN.1 would 
be the obvious ones, but i wouldn't advise using either of them, as 
they're abominations. JSON would be a good choice:

http://www.json.org/

If it's expressive enough for your objects. This is a stunningly simple 
format, and there are libraries for working with it for a wide range of 
languages.

> Are there any common mistakes that programmers do?

The key one, i'd say, is not thinking about the future. Make sure your 
protocol is able to grow - use a version number, so peers can figure out 
what language they're talking, and perhaps an option negotiation 
mechanism, if you're doing anything complex enough to warrant it (hey, you 
could always start without it and add it in a later version!). Try to 
allow for addition of new commands, message types or whatever, and for 
extension of existing ones (within reason).

> Is there a howto where I can read more about this?

Not really - protocol design is a bit of a black art. Someone asked about 
this on comp.protocols.tcp-ip a while ago:

http://groups.google.co.uk/group/comp.protocols.tcp-ip/browse_thread/thread/39f810b43a6008e6/72ca111d67768b83

And didn't get much in the way of answers. Someone did point to this, 
though:

http://www.internet2.edu/~shalunov/writing/protocol-design.html

Although i don't agree with much of what that says.

tom

-- 
limited to concepts that are meta, generic, abstract and philosophical --
IEEE SUO WG