[Chicago] threading is slow

[Oren Livne]

For a purely computational task, multiprocessing seems to give twice 
smaller speedup than the # processors in the machine: 2x for 4-proc and 
10x for 24-proc. Is that normal?

Thanks!
Oren

4-proc machine:
Serial  : time elapsed: 8.26, result = 18000068.322155
  1 procs: time elapsed: 8.35 (1.0x), result = 18000068.322155
  2 procs: time elapsed: 5.15 (1.6x), result = 18000068.322155
  4 procs: time elapsed: 4.56 (1.8x), result = 18000068.322155
  8 procs: time elapsed: 4.80 (1.7x), result = 18000068.322155

24-proc machine
Serial  : time elapsed: 12.67, result = 18000068.322155
  1 procs: time elapsed: 12.74 (1.0x), result = 18000068.322155
  2 procs: time elapsed: 7.36 (1.7x), result = 18000068.322155
  4 procs: time elapsed: 3.76 (3.4x), result = 18000068.322155
  8 procs: time elapsed: 2.42 (5.2x), result = 18000068.322155
16 procs: time elapsed: 1.31 (9.7x), result = 18000068.322155
24 procs: time elapsed: 1.31 (9.6x), result = 18000068.322155
30 procs: time elapsed: 1.33 (9.5x), result = 18000068.322155


'''
============================================================
http://stackoverflow.com/questions/4413821/multiprocessing-pool-example

Created on Mar 6, 2013
@author: Oren Livne <livne at uchicago.edu>
============================================================
'''
from multiprocessing import Pool
from time import time
import numpy as np

K = 2000000
def CostlyFunction((z,)):
     r = 0
     for k in xrange(1, K + 2):
         r += z ** (1 / k ** 1.5)
     return r

if __name__ == "__main__":
     currtime = time()
     N = 10
     w = sum(map(CostlyFunction, ((i,) for i in xrange(N))))
     t = t = time() - currtime
     print 'Serial  : time elapsed: %.2f, result = %f' % (t, w)

     for p in [1,2,4,8,16,24,30]:#2 ** np.arange(4):
         currtime = time()
         po = Pool(processes=p)
         res = po.map_async(CostlyFunction, ((i,) for i in xrange(N)))
         w = sum(res.get())
         tp = time() - currtime
         print '%2d procs: time elapsed: %.2f (%.1fx), result = %f' % 
(p, tp, t/tp, w)

On 3/6/2013 5:12 PM, Daniel Peters wrote:
> Hey Oren, if you take half an hour (or less) and pick one of these 
> videos, I have a feeling you'll get everything you need on either 
> threading or multiprocessing, or any other libs/frameworks used for 
> concurrency/parallelism .  The first listed video is even from Chipy!
>
> http://pyvideo.org/search?models=videos.video&q=threading
>
> On Wed, Mar 6, 2013 at 5:05 PM, Daniel Griffin <dgriff1 at gmail.com 
> <mailto:dgriff1 at gmail.com>>wrote:
>
>     Python has a GIL so threads mostly sort of suck. Use
>     multiprocessing, twisted or celery.
>
>
>     On Wed, Mar 6, 2013 at 3:29 PM, Oren Livne <livne at uchicago.edu
>     <mailto:livne at uchicago.edu>>wrote:
>
>         Dear All,
>
>         I am new to python multithreading. It seems that using
>         threading causes a slow down with more threads rather than a
>         speedup. should I be using the multiprocessing module instead?
>         Any good examples for threads reading from a queue with
>         multiprocessing?
>
>         Thanks so much,
>         Oren
>
>         #!/usr/bin/env python
>         '''Sum up the first 100000000 numbers. Time the speed-up of
>         using multithreading.'''
>         import threading, time, numpy as np
>
>         class SumThread(threading.Thread):
>             def __init__(self, a, b):
>                 threading.Thread.__init__(self)
>                 self.a = a
>                 self.b = b
>                 self.s = 0
>
>             def run(self):
>                 self.s = sum(i for i in xrange(self.a, self.b))
>
>         def main(num_threads):
>             start = time.time()
>             a = map(int, np.core.function_base.linspace(0, 100000000,
>         num_threads + 1, True))
>             # spawn a pool of threads, and pass them queue instance
>             threads = []
>             for i in xrange(num_threads):
>                 t = SumThread(a[i], a[i + 1])
>                 t.setDaemon(True)
>                 t.start()
>                 threads.append(t)
>
>             # Wait for all threads to complete
>             for t in threads:
>                 t.join()
>
>             # Fetch results
>             s = sum(t.s for t in threads)
>             print '#threads = %d, result = %10d, elapsed Time: %s' %
>         (num_threads, s, time.time() - start)
>
>         for n in 2 ** np.arange(4):
>             main(n)
>
>         Output:
>         #threads = 1, result = 4999999950000000, elapsed Time:
>         12.3320000172
>         #threads = 2, result = 4999999950000000, elapsed Time:
>         16.5600001812  ???
>         #threads = 4, result = 4999999950000000, elapsed Time:
>         16.7489998341  ???
>         #threads = 8, result = 4999999950000000, elapsed Time:
>         16.6720001698  ???
>
>         _______________________________________________
>         Chicago mailing list
>         Chicago at python.org <mailto:Chicago at python.org>
>         http://mail.python.org/mailman/listinfo/chicago
>
>
>
>     _______________________________________________
>     Chicago mailing list
>     Chicago at python.org <mailto:Chicago at python.org>
>     http://mail.python.org/mailman/listinfo/chicago
>
>
>
>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago


-- 
A person is just about as big as the things that make him angry.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20130307/b494cca6/attachment.html>