Parallelization in Python 2.6

[sturlamolden]

On 18 Aug, 13:45, Robert Dailey <rcdai... at gmail.com> wrote:

> Really, all I'm trying to do is the most trivial type of
> parallelization. Take two functions, execute them in parallel. This
> type of parallelization is called "embarrassingly parallel", and is
> the simplest form. There are no dependencies between the two
> functions.

If you are using Linux or Mac, just call os.fork for this.

You should also know that you function "create_task" is simply

from threading import Thread
def create_task(task):
   Thread(target=task).start()

If your task releases the GIL, this will work fine.


> They do requires read-only access to shared data, though.
> And if they are being spawned as sub-processes this could cause
> problems, unless the multiprocess module creates pipelines or other
> means to handle this situation.

With forking or multiprocessing, you have to use IPC. That is, usually
pipes, unix sockets / named pipes, or shared memory. Multiprocessing
helps you with this. Multiprocessing also has a convinient Queue
object for serialised read/write access to a pipe.

You can also create shared memory with mmap.mmap, using fd 0 with
Windows or -1 with Linux.