Optimizing sockets to 1000 requests/sec?

[Skip Montanaro]

    Paul> If it's on the local machine, why mess with TCP at all, instead of
    Paul> using AF_UNIX?  I'm not sure whether the OP intended to run
    Paul> locally tough.

Performance difference isn't that great, and AF_INET obviously lets you
migrate the processes to separate machines if you need to.  Coincidentally,
a few nights ago was doing a little "what if" testing in this area.  With
two processes on the same machine (my laptop is a 450MHz P-III)
communicating via unix sockets and datagrams I was able to push about 5000
24-byte msgs/sec one-way.  When I switched to AF_INET and stream
connections, the transfer rate only dropped to about 4700 msgs/sec.

Last night I was messing around with my Python compilation options and as a
result I've changed my build options.  I'm now using

    --with-threads
    --without-pymalloc
    OPT=-O3 -minline-all-stringops -fomit-frame-pointer

I just measured my TCP/stream rate at about 5200 msgs/sec and unix/datagram
rate at about 6400 msgs/sec.  Seems like the inline-all-stringops flag
really helped (it had a modest effect on pystones, but it doesn't push a lot
of strings around - my little message-passing test does).  I'm a bit
mystified by the different effects on the two versions though.  The only
difference in the code is in the type of socket connection.  The rest of the
code is identical.

we're-all-slinging-numbers-but-noone's-showing-their-cards-ly, y'rs,

-- 
Skip Montanaro (skip at pobox.com - http://www.mojam.com/)
It's only a 25% solution to our problems.  Of course, we only want to solve
25% of our problems, so it becomes a 100% solution.