Catching exceptions with multi-processing

[Chris Angelico]

On Sat, Jun 20, 2015 at 1:41 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Sat, 20 Jun 2015 12:01 am, Fabien wrote:
>
>> Folks,
>>
>> I am developing a tool which works on individual entities (glaciers) and
>> do a lot of operations on them. There are many tasks to do, one after
>> each other, and each task follows the same interface:
>
> I'm afraid your description is contradictory. Here you say the tasks run one
> after another, but then you say:
>
>> This way, the tasks can be run in parallel very easily:
>
> and then later still you contradict this:
>
>> Also, the task2 should not be run if task1 threw an error.
>
>
> If task2 relies on task1, then you *cannot* run them in parallel. You have
> to run them one after the other, sequentially.

AIUI what he's doing is all the subparts of task1 in parallel, then
all the subparts of task2:

pool.map(task1, dirs, chunksize=1)
pool.map(task2, dirs, chunksize=1)
pool.map(task3, dirs, chunksize=1)

task1 can be done on all of dirs in parallel, as no instance of task1
depends on any other instance of task1; but task2 should be started
only if all task1s finish successfully. OP, is this how it is? If not,
I apologize for the noise.

ChrisA