multiprocessing shows no benefit

[Ian Kelly]

On Wed, Oct 18, 2017 at 10:21 AM, Jason <jasonhihn at gmail.com> wrote:
> On Wednesday, October 18, 2017 at 12:14:30 PM UTC-4, Ian wrote:
>> On Wed, Oct 18, 2017 at 9:46 AM, Jason  wrote:
>> > #When I change line19 to True to use the multiprocessing stuff it all slows down.
>> >
>> > from multiprocessing import Process, Manager, Pool, cpu_count
>> > from timeit import default_timer as timer
>> >
>> > def f(a,b):
>> >         return dict_words[a]-b
>>
>> Since the computation is so simple my suspicion is that the run time
>> is dominated by IPC, in other words the cost of sending objects back
>> and forth outweighs the gains you get from parallelization.
>>
>> What happens if you remove dict_words from the Manager and just pass
>> dict_words[a] across instead of just a? Also, I'm not sure why
>> dict_keys is a managed list to begin with since it only appears to be
>> handled by the main process.
>
> You can try that code by changing line 17 :-) It's 10 times faster.
>
> Well given the many objects to be iterated over, I was hoping pool.map() would distribute them across the processors. So that each processor gets len(dict_words)/cpu_count() items to process. The actual computation is much longer than a single subtraction, currently I can process about 16k items per second single core.  My target is to get those 16k items processed in .25s.

Well, in any case I would try keeping the communication to a minimum.
Instead of sending 1 item at a time, break the data set up into
batches of 4k and send each child task a batch. Avoid using managers
or other shared state.