Catching exceptions with multi-processing

[Jean-Michel Pichavant]

----- Original Message -----
> From: "Fabien" <fabien.maussion at gmail.com>
> To: python-list at python.org
> Sent: Friday, 19 June, 2015 4:01:02 PM
> Subject: Catching exceptions with multi-processing
> 
> Folks,
> 
> I am developing a tool which works on individual entities (glaciers)
> and
> do a lot of operations on them. There are many tasks to do, one after
> each other, and each task follows the same interface:
> 
> def task_1(path_to_glacier_dir):
>      open file1 in path_to_glacier_dir
>      do stuff
>      if dont_work:
>          raise RuntimeError("didnt work")
>      write file2 in path_to_glacier_dir
> 
> This way, the tasks can be run in parallel very easily:
> 
> import multiprocessing as mp
> pool = mp.Pool(4)
> 
> dirs = [list_of_dirs]
> pool.map(task1, dirs, chunksize=1)
> pool.map(task2, dirs, chunksize=1)
> pool.map(task3, dirs, chunksize=1)
> 
> ... and so forth. I tested the tool for about a hundred glaciers but
> now
> it has to run for thousands of them. There are going to be errors,
> some
> of them are even expected for special outliers. What I would like the
> tool to do is that in case of error, it writes the identifier of the
> problematic glacier somewhere, the error encountered and more info if
> possible. Because of multiprocessing, I can't write in a shared file,
> so
> I thought that the individual processes should write a unique "error
> file" in a dedicated directory.
> 
> What I don't know how to, however, is how to do this at minimal cost
> and
> in a generic way for all tasks. Also, the task2 should not be run if
> task1 threw an error. Sometimes (for debugging), I'd rather keep the
> normal behavior of raising an error and stopping the program.
> 
> Do I have to wrap all tasks with a "try: exept:" block? How to switch
> between behaviors? All the solutions I could think about look quite
> ugly
> to me. And it seems that this is a general problem that someone
> cleverer
> than me had solved before ;-)
> 
> Thanks,
> 
> Fabien

https://azylstra.net/blog/post/exception-handling-with-python-processes

shows how subprocesses can send their exception to the main process.



-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.