Multiprocessing taking too much time

[Dan Stromberg]

On Thu, Jul 29, 2010 at 12:04 PM, John Nagle <nagle at animats.com> wrote:

> On 7/29/2010 11:08 AM, Shailendra wrote:
>
>> Hi All,
>> I have a following situation.
>> ==================PSUDO CODE START==================
>> class holds_big_array:
>>     big_array  #has a big array
>>
>>     def get_some_element(self, cond) # return some data from the array
>> from the big array
>> ==================PSUDO CODE END====================
>> I wanted to use multiprocessing module to parallelise calling
>> "get_some_element". I used following kind of code
>>
>> ==================PSUDO CODE START==================
>> pool = Pool(processes=2)
>> holder =holds_big_array() #class instantiation
>> def callback_f(result):
>>          do something with result
>> loop many times
>>    pool.apply_async(holder.get_some_element,args,callback=callback_f)
>> pool.close()
>> pool.join()
>> ==================PSUDO CODE END====================
>> Note: Had to do something to enable instance method being pickled...
>>
>> I tested this with less than realistic size of big_array . My parallel
>> version works much slower than than the normal serial version (10-20
>> sec vs 7-8 min). I was wonder what could be the possible reason.\
>
>
I think the place to start when code is slow, is usually to use a profiler.

However, in this case it's likely as you say, a pickling issue.


>    It's hard to tell from your "PSUDO CODE", but it looks like each
> access to the "big array" involves calling another process.
>
>    Calling a function in another process is done by creating an
> object to contain the request, running it through "pickle" to convert
> it to a stream of bytes, sending the stream of bytes through a socket or
> pipe to the other process, running the byte stream through "unpickle" to
> create an object like the original one, but in a different process, and
> calling a function on the newly created object in the receiving process.
>  This entire sequence has to be done again in reverse
> to get a reply back.
>
>    This is hundreds of times slower than a call to a local function.
>
>    The "multiprocessing module" is not a replacement for thread-level
> parallelism.  It looks like it is, but it isn't.  It's only useful for
> big tasks which require large amounts of computation and little
> interprocess communication.  Appropriately-sized tasks to send out
> to another process are things like "parse large web page" or
> "compress video file", not "access element of array".


Well, multiprocessing'll replace threading in many scenarios, including some
where CPython's threading isn't very useful.

The O.P. might look into storing their array in shared memory -
multiprocessing facilitates that pretty well.  Or, as John has suggested,
chunk up the work into larger pieces than individual array accesses.

HTH.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20100730/ff347817/attachment-0001.html>