[Python-ideas] reducing multiprocessing.Queue contention

[Eli Bendersky]

On Wed, Jan 23, 2013 at 3:16 AM, Charles-François Natali <
cf.natali at gmail.com> wrote:

> Hello,
>
> Currently, multiprocessing.Queue put() and get() methods hold locks
> for the entire duration of the writing/reading to the backing
> Connection (which can be a pipe, unix domain socket, or whatever it's
> called on Windows).
>
> For example, here's what the feeder thread does:
> """
>            else:
>                wacquire()
>                try:
>                    send(obj)
>                    # Delete references to object. See issue16284
>                    del obj
>                finally:
>                    wrelease()
> """
>
> Connection.send() and Connection.recv() have to serialize the data
> using pickle before writing them to the underlying file descriptor.
> While the locking is necessary to guarantee atomic read/write (well,
> it's not necessary if you're writing to a pipe less than PIPE_BUF, and
> writes seem atomic on Windows), the locks don't have to be held while
> the data is serialized.
>
> Although I didn't make any measurement, my gut feeling is that this
> serializing can take a non negligible part of the overall
> sending/receiving time, for large data items. If that's the case, then
> simply holding the lock for the duration of the read()/write() syscall
> (and not during serialization) could reduce contention in case of
> large data sending/receiving.
>
> One way to do that would be to refactor the code a bit to provide
> maybe a (private) AtomicConnection, which would encapsulate the
> necessary locking: another advantage is that this would hide the
> platform-dependent code inside Connection (right now, Queue only uses
> a lock for ending on Unix platforms, since write is apparently atomic
> on Windows).
>
>
In general, this sounds good. There's indeed no reason to perform the
serialization under a lock.

It would be great to have some measurements to see just how much it takes,
though.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130123/4f003538/attachment.html>