Simple TCP proxy

[Barry Scott]

Morten,

As Chris remarked you need to learn a number of networking, python, system performance
and other skills to turn your project into production code.

Using threads does not scale very well. Its uses a lot of memory and raises CPU used
just to do the context switches. Also the GIL means that even if you are doing blocking
I/O the use of threads does not scale well.

Its rare to see multi threaded code, rather what you see is code that uses async I/O.

At its heart async code at the low level is using a kernel interface like epoll (or on old
systems select). What epoll allow you to do is wait on a sets of FDs for a range of
I/O operations. Like ready to read, ready to write and other activity (like the socket
closing).

You could write code to use epoll your self, but while fun to write you need to know
a lot about networking and linux to cover all the corner cases.

Libraries like twisted, trio, uvloop and pythons selectors implemented production quality
version of the required code with good APIs.

Do not judge these libraries by their size. They are no bloated and only as complex as
the problem they are solving requires.

There is a simple example of async code using the python selectors here that shows
the style of programming.
https://docs.python.org/3/library/selectors.html#examples <https://docs.python.org/3/library/selectors.html#examples>

The issues that you likely need to solve and test for include:
* handling unexpected socket close events.
* buffering and flow control from one socket's read to the another socket's write.
  What if one side is reading slower then the other is writing?
* timeout sockets that stop sending data and close them

At some point you will exceed the capacity for one process to handle the load.
The solution we used is to listen on the socket in a parent process and fork
enough child processes to handle the I/O load. This avoids issues with the GIL
and allows you to scale.

But I am still not sure why you need to do anything more the increase the backlog
on your listen socket in the main app. Set the backlog to 1,000,000 does that fix
your issue? 

You will need on Linux to change kernel limits to allow that size. See man listen
for info on what you need to change.

Barry