parallel programming in Python

Jabba Laci jabba.laci at gmail.com
Thu May 10 08:14:47 EDT 2012


Hi,

I would like to do some parallel programming with Python but I don't
know how to start. There are several ways to go but I don't know what
the differences are between them: threads, multiprocessing, gevent,
etc.

I want to use a single machine with several cores. I want to solve
problems like this: iterate over a loop (with millions of steps) and
do some work at each step. The steps are independent, so here I would
like to process several steps in parallel. I want to store the results
in a global list (which should be "synchronised"). Typical use case:
crawl webpages, extract images and collect the images in a list.

What's the best way?

Thanks,

Laszlo



More information about the Python-list mailing list