Comms Protocol design dilemma

[Jp Calderone]

On Mon, Apr 07, 2003 at 11:37:23PM +0530, Jeethu Rao wrote:
> Hi,
> I've been working on a encrypted instant messenger server for
> quite some time now. I've had to dump the code about 3 times
> (I changed the protocols). And quite frankly I don't feel that
> bad while throwing out code written in python, since its very
> easy to rewrite the same functionality (Throwing away *hard written*
> C code really breaks my heart.)

  Do you mean you've dumped the entire server, not just the protocol
handling code?  If so, it sounds like you could be doing a little better job
keeping these separate areas of your code actually separate.

> 
> Now all the past three implementations are based on a packet
> Oriented scheme <TCP/IP, Blowfish Encryption,16 byte packet hdr>.
> My friends in here are suggesting me to use a line oriented protocol
> (Like Microsoft Messenger), some have even suggested using XML
> (Which I vehemently hate) like in Jabber. I'm of the opinion that
> XML tends to waste a lot of bandwidth.

  I agree.  Don't use XML.  Additionally, since you're not sending
plain-text, and so you're not sending lines, I don't think you should use a
line-oriented protocol.

> 
> Since most of the data is binary, If I go in for line oriented protocol,
> I'll have to base64 (or something similar) all the encrypted data,
> Which will be an added overhead.

  Correct.

> 
> The primary dilemma is, What protocol is better suited, 'text based' or
> 'packet based' ?

  It seems pretty obvious that a text based protocol isn't better suited. 
Maybe if you were actually sending textual data...

> 
> The other problem is in the implementation.
> The old implementations used a simple scheme, every incoming
> Connection would be handled by a thread from a thread pool.

  This sounds like a bad idea.  How can it scale to many thousands (a small
number, given the application, right?) of users this way?

> But now, the client is going to be connected to the server
> All the time its online. Thus, a lot of clients means a lot
> Of threads running in parallel, doing nothing. I'm not quite sure
> If the Python threads waiting on sockets would yield as soon
> as it gets its share of the VM (They're all serialized, right ?).
> And since there's a lot of encryption/ decryption , hashing, random
> number generation involved, I can pretty much rule out asynchronous
> sockets. Is there any better strategy to tackle this problem?
> 

  Ahh, I see, in the previous scheme, "connections" (maybe you were even
using UDP) were short lived, so one thread might handle many of them over a
short period of time.

  I don't think you need to rule out the asynchronous solution.  Check out
Twisted.  You can have all your network code run in a single thread, and all
the encryption/decryption/hashing/random number generation run in any number
of worker threads.  Here's how easy it might be (untested code):

    from twisted.internet.protocol import Protocol
    from twisted.internet.threads import deferToThread

    from Crypto.PublicKey import RSA

    class MessageProtocol(Protocol):
        buffer = ''
        def dataReceived(self, data):
            self.buffer += data
            if len(self.buffer) >= PACKET_SIZE:
                buf = self.buffer[:PACKET_SIZE]
                self.buffer = self.buffer[PACKET_SIZE:]
                d = deferToThread(rsaInstance.decrypt, buf)
                d.addCallback(self.decryptedDataReceived)
                d.addErrback(self.decryptionFailed)

        def decryptedDataReceived(self, data):
            """Override me"""

        def decryptionFailed(self, failure):
            """Override me"""

        def sendData(self, data):
            d = deferToThread(rsaInstance.encrypt, data)
            d.addCallback(self.sendEncrypted)
            return d

        def sendEncrypted(self, data):
            self.transport.write(data)

  Twisted manages the thread pool for you, takes care of getting things
called in the right thread, and lets you use just one thread for network IO.

  Check it out - http://www.twistedmatrix.com/

  Jp

-- 
Examinations are formidable even to the best prepared, for
even the greatest fool may ask more the the wisest man can answer.
                -- C.C. Colton
-- 
 up 18 days, 15:01, 12 users, load average: 0.12, 0.09, 0.06