Shutting down a cross-platform multithreaded app

[James Harris]

"Chris Angelico" <rosuav at gmail.com> wrote in message 
news:mailman.13.1442657702.21674.python-list at python.org...
> On Sat, Sep 19, 2015 at 7:49 PM, James Harris 
> <james.harris.1 at gmail.com> wrote:
>> "Chris Angelico" <rosuav at gmail.com> wrote in message
>> news:mailman.8.1442612439.21674.python-list at python.org...

...

>>> If you're using select() to monitor the sockets, you don't actually
>>> then have to _do_ anything with the shutdown socket. You could have 
>>> a
>>> single socket that sends the shutdown signal to all your workers.
>>
>>
>> I don't understand how a single socket could send the signal to all 
>> the
>> workers. I did consider some form of multicast but thought it too
>> complicated (and possibly infeasible).
>
> The way I'm describing it, the workers never actually read from the
> socket. Once that socket becomes readable, they immediately shut down,
> without making the socket no-longer-readable.

Understood. Good idea. Initial thoughts on it: Would work for threads. 
Would save on the number of sockets required. Would not work for 
processes if the model was ever changed. Would make it easier for rogue 
packets to shut a worker down. Would not allow any way to distinguish 
between shutdown priorities (not something I have mentioned). Definitely 
feasible. I'll keep it in mind.

>>> Bear in mind, though, that Windows has no protection against other
>>> processes shutting you down. You can restrict it to 127.0.0.1 (of
>>> course) but any program running on the same computer as the server -
>>> regardless of user permissions etc - will be able to connect to your
>>> sockets.

...

>> That sounds similar to what I had in mind but I am not sure why you 
>> would
>> close the listening socket. Connections could come in at any time and
>> threads could therefore be needed at any time so I was thinking that 
>> the
>> master thread (the one with the listening TCP socket) would just sit 
>> waiting
>> for new connection requests (or an interrupting signal).
>
> TCP sockets work on the basis of a master socket and any number of
> spawned sockets. The master is what gives you an open port; each
> spawned socket represents one connection with one client. Once you
> have an established connection, the master should be able to be closed
> without disrupting that. No other process will be able to connect to
> you, but you'll still be able to use one end of the socket to make the
> other end readable.

Agreed but I need the listening socket to remain open and listening for 
new connections (at least until the whole program is told to shut down).

>>> This will make it difficult for ordinary userspace code to mess with
>>> you. It'd still be possible, I think, for something with raw sockets
>>> access to feign a termination signal; I have no idea what 
>>> protections
>>> Windows offers you there.
>>
>>
>> Yes, something which could forge a packet could tell a worker to 
>> close down.
>> I don't think there's any significant problem here, thought, because:
>>
>> * other programs are similarly vulnerable to forged packets
>> * the only forgery effect is to tell a worker thread to stop - not a 
>> big
>> loss
>> * the shutdown protocol would/should cause the client to re-request
>> * a forger would have to know the specific port number used by the 
>> master
>> thread to communicate with that particular worker, and the port 
>> number that
>> worker was using.
>>
>> Overall, I think it would be more than robust enough.
>
> With UDP, any process that can send a UDP packet can flood the system
> with them until your workers shut down. You wouldn't even notice until
> it succeeds.

Is that true? You seem to be describing a non-forged attack but to get 
the source UDP port right wouldn't the attacker have to be runing on the 
same machine *and* to bind to the same port that the machine had 
allocated to my program? I might be wrong but I don't think the UDP 
stack would allow the same port to be bound again before the original 
had been closed.

> With TCP, at least an attacker would need raw socket
> access. It's still not as protected as a Unix domain socket, but it's
> a bit harder for someone to do.
>
>> Notwithstanding your comment about a single socket, above, no one 
>> seems so
>> far to have objected to the number of sockets this would need, which 
>> was my
>> main concern, so that's good!
>
> Sure. Sockets are pretty cheap. Even if you had one for every worker,
> there's room for you to have thousands (maybe tens of thousands) of
> workers without a problem. I think you'll run into other scaling
> problems with that many workers on one computer :)

Let's see. If I stick with my original plan then each worker would have 
a TCP socket and a UDP socket. The "listener" thread would have its 
single listening TCP socket plus it would have a UDP socket for each 
worker thread. Total of three sockets per worker, two of which would be 
UDP sockets with port numbers assigned and thus consumed.

If I go with a single "shutdown socket" then I would have just one 
socket per worker. That would use fewer sockets, for sure.

James