taking python enterprise level?...

[D'Arcy J.M. Cain]

On Thu, 25 Feb 2010 15:29:34 +0000
"Martin P. Hellwig" <martin.hellwig at dcuktec.org> wrote:
> On 02/25/10 13:58, D'Arcy J.M. Cain wrote:
> > On Thu, 25 Feb 2010 02:26:18 -0800 (PST)
> <cut>
> > Our biggest problem was in
> > a network heavy element of the app and that was low level TCP/IP stuff
> > that rather than being Python's problem was something we used Python to
> > fix.
> <cut>
> Out off interest, could you elaborate on that?

Somewhat - there is an NDA so I can't give exact details.  It was
crucial to our app that we sync up databases in Canada and the US (later
Britain, Europe and Japan) in real time with those transactions.  Our
problem was that even though our two server systems were on the
backbone, indeed with the same major carrier, we could not keep them in
sync.  We were taking way to long to transact an update across the
network.

The problem had to do with the way TCP/IP works, especially closer to
the core.  Our provider was collecting data and sending it only after
filling a buffer or after a timeout.  The timeout was short so it
wouldn't normally be noticed and in most cases (web pages e.g.) the
connection is opened, data is pushed and the connection is closed so
the buffer is flushed immediately.  Our patterns were different so we
were hitting the timeout on every single transaction and there was no
way we would have been able to keep up.

Our first crack at fixing this was to simply add garbage to the packet
we were sending.  Making the packets an order of magnitude bigger sped
up the proccessing dramatically.  That wasn't a very clean solution
though so we looked for a better way.

That better way turned out to asynchronous update transactions.  All we
did was keep feeding updates to the remote site and forget about ACKS.
We then had a second process which handled ACKS and tracked which
packets had been properly transferred.  The system had IDs on each
update and retries happened if ACKS didn't happen soon enough.
Naturally we ignored ACKS that we had already processed.

All of the above (and much more complexity not even discussed here) was
handled by Python code and database manipulation.  There were a few
bumps along the way but overall it worked fine.  If we were using C or
even assembler we would not have sped up anything and the solution we
came up with would have been horrendous to code.  As it was I and my
chief programmer locked ourselves in the boardroom and had a working
solution before the day was out.

Python wins again.

-- 
D'Arcy J.M. Cain <darcy at druid.net>         |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 425 1212     (DoD#0082)    (eNTP)   |  what's for dinner.