thread, multiprocessing: communication overhead

[Aaron Brady]

On Dec 30, 9:46 am, mk <mrk... at gmail.com> wrote:
> Hello everyone,
>
> This time I decided to test communication overhead in multithreaded /
> multiprocess communication. The results are rather disappointing, that
> is, communication overhead seems to be very high. In each of the
> following functions, I send 10,000 numbers to the function / 10 threads
> / 10 processes, which simply returns it in its respective way.
>
> Function: notfun            Best: 0.00622 sec   Average: 0.00633 sec
> (simple function)
>
> Function: threadsemfun      Best: 0.64428 sec   Average: 0.64791 sec
> (10 threads synchronizing using semaphore)
>
> Function: threadlockfun     Best: 0.66288 sec   Average: 0.66453 sec
> (10 threads synchronizing using locks)
>
> Function: procqueuefun      Best: 1.16291 sec   Average: 1.17217 sec
> (10 processes communicating with main process using queues)
>
> Function: procpoolfun       Best: 1.18648 sec   Average: 1.19577 sec
> (a pool of 10 processes)
>
> If I'm doing smth wrong in the code below (smth that would result in
> performance suffering), please point it out.
snips
> def threadsemfun():
>          sem = threading.Semaphore()
> def threadlockfun():
>          sem = threading.Semaphore()

You used a Semaphore for both lock objects here.

'multiprocessing' is a really high level layer that makes a lot of
decisions about trade-offs, has highly redundant communication, and is
really easy to use.  If you want to save a byte, you'll have to make
your own decisions about trade-offs and redundancies (possibly even
looking at real result data to make them).

I actually think 'multiprocessing' is really good, and even if I hand-
wrote my own IPC, it would be slower!

CMIIW, but I believe your timing function includes the time to launch
the actual processes and threads, create the synch. objects, etc.  You
might try it again, creating them first, starting the timer, then
loading them.