[Python-ideas] reducing multiprocessing.Queue contention

[Charles-François Natali]

> But you can only rely on the atomicity of writing less than PIPE_BUF bytes
> if you know that no other process is currently trying to send a message
> longer than PIPE_BUF.  Otherwise the short message could be embedded in the
> long message (even if the process sending the long message is holding the
> lock).

Maybe I wasn't clear.
I'm not suggesting to not hold the lock when sending less than
PIPE_BUF, since it wouldn't work in the case you describe above.
I'm suggesting to serialize the data prior to acquiring the writer
lock, to reduce contention (and unserialize after releasing the
reading lock).

(I only mentioned PIPE_BUF because I was sad to see that Windows
supported atomic messages, and this comforted me a bit :-)