[SciPy-Dev] BSP interface in the SciPy.

[Sturla Molden]

On 13.02.2012 06:56, Edward J. Yoon wrote:

> Today, I just noticed that there's a BSP interface based on BSPLib in
> the SciPy, and thought maybe we could work together, on supporting
> SciPy programs to run on existing Hadoop YARN[3] or Hama cluster.

AFAIK, BSP is a coding style, not a particular API.

If you need a barrier for BSP synchronization, this is the simplest 
implementation I can think of:


from multiprocessing import Event
from math import ceil, log
from contextlib import contextmanager

def _barrier(b):
     @contextmanager
     def _context(rank):
         b.wait(rank)
         yield
         b.wait(rank)
     return _context

class Barrier(object):

     def __init__(self, numproc):
         self._events = [mp.Event() for n in range(numproc**2)]
         self._numproc = numproc
         self.barrier = _barrier(self)

     def wait(self, rank):
         # loop log2(numproc) times, rounding up
         for k in range(int(ceil(log(self._numproc)/log(2)))):

             # send event to process
             # (rank + 2**k) % numproc
             receiver = (rank + 2**k) % self._numproc
             evt = self._events[rank * self._numproc + receiver]
             evt.set()

             # wait for event from process
             # (rank - 2**k) % numproc
             sender = (rank - 2**k) % self._numproc
             evt = self._events[sender * self._numproc + rank]
             evt.wait()
             evt.clear()



Now BSP code could look like this:


    barrier = Barrier(numprocs)

    for data in container:

        <process data>

        with barrier.barrier(rank):

             <communicate>




Sturla