Problem with blocking portably on sockets and Queue?

[Alex Martelli]

<posted & mailed>

Tero Saarni wrote:

> Hi,
> 
> I have several threads communicating with each other using events
> stored in Queues. Threads block on Queue.get() until somebody
> publishes an event in thread's event queue.

So far so wonderful -- excellent architecture.


> I need to add support for sockets to the system. Thread needs to
> unblock when:
> 
>   - there is socket ready to be read, or
>   - there is event waiting in the queue
> 
> My first tought was to replace blocking on Queue.get() with blocking
> on poll or select and dedicating file descriptors (created with os.pipe())
> as an semaphore. Event publisher would write something to the write
> end of the pipe when it puts an event to the queue, effectively
> unblocking the receiver.
> 
> BUT I noticed that select.poll() is not available on Windows and
> secondly Windows version of select.select() will accept only socket
> descriptors.
> 
> What options do I have that are still portable also to Windows
> platform?

I think you could devote a thread specifically to the task of handling
sockets, only.  That special thread would not block on any queue but
just on select; when its select shows that a socket is ready, you can
have the thread itself do the reading and post the appropriate special
message including the data read to the appropriate thread, or you could
pass the ready-for-reading socket object to the appropriate thread, or
the like.  This needs a bit more care if the set of sockets to be
select'ed on changes with time; in this case, I think the socket-handler
thread would have to use a timeout on its select so the set of sockets
can be updated once in a while (either by another thread, in which
case you need a lock to protect the "set of sockets" shared data, or
by the socket-handling thread (SHT) itself in response to a queued 
message -- the SHT would read non-blockingly from that queue periodically
when the select times out) -- or you could perhaps use a special
identifiable socket for the SHT to receive from other threads requests 
to change the set of ('true') sockets to be selected on.


Alex