Pickle based workflow - looking for advice

[Chris Angelico]

On Tue, Apr 14, 2015 at 11:08 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> On Tue, 14 Apr 2015 05:58 pm, Fabien wrote:
>
>> On 14.04.2015 06:05, Chris Angelico wrote:
>>> Not sure what you mean, here. Any given file will be written by
>>> exactly one process? No possible problem. Multiprocessing within one
>>> application doesn't change that.
>>
>> yes that's what I meant. Thanks!
>
> It's not that simple though. If you require files to be written in precisely
> a certain order, then parallel processing requires synchronisation.
>
> Suppose you write A, then B, then C, then D, each in it's own process (or
> thread). So the B process has to wait for A to finish, the C process has to
> wait for B to finish, and so on. Otherwise you could find yourself with C
> reading the data from B before B is finished writing it.

Sure, which is a matter of writer/reader conflicts on a single file -
nothing to do with "writing multiple files simultaneously" which was
the question raised.

ChrisA