thread, multiprocessing: communication overhead

[mk]

Aaron Brady wrote:

> snips
>> def threadsemfun():
>>          sem = threading.Semaphore()
>> def threadlockfun():
>>          sem = threading.Semaphore()
> 
> You used a Semaphore for both lock objects here.

Right... I corrected that (simply changed to threading.Lock() in 
threadlockfun) and the result is much better, though still an order of 
magnitude worse than plain function:

Function: threadlockfun     Best: 0.08665 sec   Average: 0.08910 sec
Function: notfun            Best: 0.00987 sec   Average: 0.01003 sec


> 'multiprocessing' is a really high level layer that makes a lot of
> decisions about trade-offs, has highly redundant communication, and is
> really easy to use.  If you want to save a byte, you'll have to make
> your own decisions about trade-offs and redundancies (possibly even
> looking at real result data to make them).

Hmm, do you think that lower-level 'thread' module might work more 
efficiently?

> I actually think 'multiprocessing' is really good, and even if I hand-
> wrote my own IPC, it would be slower!
> 
> CMIIW, but I believe your timing function includes the time to launch
> the actual processes and threads, create the synch. objects, etc.  You
> might try it again, creating them first, starting the timer, then
> loading them.

Except I don't know how to do that using timeit.Timer. :-/