Pickle based workflow - looking for advice

Tue Apr 14 09:08:46 EDT 2015

On Tue, 14 Apr 2015 05:58 pm, Fabien wrote:

> On 14.04.2015 06:05, Chris Angelico wrote:
>> Not sure what you mean, here. Any given file will be written by
>> exactly one process? No possible problem. Multiprocessing within one
>> application doesn't change that.
> 
> yes that's what I meant. Thanks!

It's not that simple though. If you require files to be written in precisely
a certain order, then parallel processing requires synchronisation.

Suppose you write A, then B, then C, then D, each in it's own process (or
thread). So the B process has to wait for A to finish, the C process has to
wait for B to finish, and so on. Otherwise you could find yourself with C
reading the data from B before B is finished writing it.

-- 
Steven